1 Introduction

Models addressing open questions concerning the standard model of particle physics (SM) usually resort to the use of new symmetries and/or the addition of extra particles. As a first example, we can mention models implementing different see-saw mechanisms (type I, II and III) which introduce bosonic or fermionic degrees of freedom in order to explain tiny neutrino masses and their mixings [1,2,3,4]. In combination with that, new abelian and non-abelian symmetries are also invoked in order to obtain highly predictive scenarios where not only neutrino masses are fixed but also further correlations between neutrino oscillation parameters appear [5,6,7,8,9,10,11,12,13,14,15,16]. A series of models with a matter content larger than the one of the SM are those dealing with the impressive observation that almost thirty percent of the energy content of the Universe is due to dark matter (DM) [17]. Arguably, the simplest model providing a DM candidate is that which extends the SM only by a real scalar transforming in a non trivial way under a stabilizing \({\mathbb {Z}}_2\) discrete symmetry [18, 19]. However, other well-motivated models based on supersymmetry [20,21,22,23], extra dimensions [24], the \(B-L\) symmetry [25,26,27,28] and those with an Axion/ALP [29,30,31,32,33,34,35] have been widely considered (see [36] for a review). All of them have additional symmetries and extra particles in their physical spectrum.

In the same vein, the so-called \(3-3-1\) models are interesting extensions of the SM. The fundamental idea behind all these models is to extend the \({\mathrm {SU}}(2)_{L}\) gauge group to the \(\text {SU}(3)_{L}\) one, in other words, the total gauge group of these models is \(\text {G}_{331}\equiv \text {SU}(3)_{C}\otimes \text {SU}(3)_{L}\otimes \text {U}(1)_{N}\). Here, C and L stand, as in the SM, for color and left chirality, respectively. However, N stands for a new charge different from the SM hypercharge, Y, and its values are assigned to obtain the latter after the first spontaneous symmetry breaking. More specifically, the values of N together with an embedding parameter b determine the electric charges of the matter content in these models since the electric charge operator is \(Q=T_{3}-bT_{8}+N{\mathbf {1}}_{3\times 3}\) [37], where \(T_{3,\,8}\) are the diagonal Gell-Mann matrices, \({\mathbf {1}}_{3\times 3}\) is the \(3\times 3\) identity matrix and the parameter b can take two values: \(1/\sqrt{3}\) or \(\sqrt{3}\).

In this paper, we consider the \(3-3-1\) model with \(b=1/\sqrt{3}\) and the simplest scalar sector as proposed in Refs. [38, 39] after a systematic study of all possible \(3-3-1\) models without exotic electric charges. This model is known in the literature as the “economical \(3-3-1\) model”  and has some appealing features that turn it arguably the most interesting \(3-3-1\) model. Among these properties we can mention that right-handed neutrinos, \(N_{a}\), are in the same \(\text {SU}(3)_{L}\) multiplet as the SM leptons, \(\nu _{a}\) and \(e_{a}\). This is possible because the fundamental representation of the \(\text {SU}(3)_{L}\) gauge group is larger than the \(\text {SU}(2)_{L}\) one and the parameter \(b=1/\sqrt{3}\) allows the \(\text {SU}(3)_{L}\) multiplet to have two electrically neutral components: \(\nu _{a}\) and \(N_{a}^{c}\). This property allows for massive neutrinos at tree level. Yet, agreement with experiments is reached only when one-loop contributions to neutrino masses are considered [40]. Other no less important features of this model are the possibility of implementing the Peccei-Quinn mechanism in order to solve the strong CP problem [41] and the existence of axion dark matter [35]. Needless to say, this model also shares some appealing features with other versions [42,43,44,45,46,47,48,49] such as the capability to shed some light on the family replication issue of the SM.

Although the economical \(3-3-1\) model has several appealing features, it also introduces a considerable number of degrees freedom that turn it less predictive. For instance, its scalar potential has at least nineteen coupling constants. It is a large number when compared with two couplings in the SM scalar potential. Needless to say, a large number of extra Yukawa couplings is allowed by the \(3-3-1\) gauge group. Therefore, in this paper we search for constraints that allow to reduce this, in some sense, undesirable freedom. More specifically, we study vacuum stability at tree level, i.e. the conditions guaranteeing that the scalar potential is bounded from below in all directions in the field space as the field norms approach to infinity. It is well-known that in the SM, at tree level, it is enough to make the Higgs boson quartic coupling positive [50,51,52]. Nevertheless, in the case of the economical \(3-3-1\) model we face a more complicated problem even at tree level since we have to deal with nineteen coupling constants and the scalar fields belong to the fundamental and anti-fundamental representations of \(\text {SU}(3)_{L}\). However, the problem becomes simpler when a \({\mathbb {Z}}_2\) symmetry acting on some fields is considered. This symmetry is already used in the \(3-3-1\) literature [37, 38, 41, 53,54,55,56,57] with different motivations. In the present context, this symmetry not only reduces the number of coupling constants to fourteen, but also makes the quartic terms in the scalar potential to have a \(\lambda _{ij}\phi _i^2\phi _j^2\) form. Therefore, demanding that the scalar potential is bounded from below as the field norms approach to infinity is equivalent to ensuring that the \(\lambda _{ij}\) matrix is copositive (positive on nonnegative vectors) [58,59,60,61]. In addition, to make the problem even more tractable, we use the method of the orbit space in Refs. [62, 63] which greatly reduces the number of variables. At the end, the problem of vacuum stability is reduced to study the copositivity of a \(3\times 3\) matrix. This provides seventeen inequalities, for the ten quartic couplings, that at first sight seem too complicated in order to provide useful analytical relations. However, combining these inequalities with constraints coming from the first and second derivative tests for a minimum of the scalar potential, we manage to find enlightening analytical constraints for these coupling constants.

Finally, with the aim of restricting the rest of the scalar couplings we turn our attention to the scalar mass spectrum, since all of the squared scalar masses must be positive in general. However, these masses also depend on the vacuum expectation values (VEVs) of the scalar fields. Thus, we use the experimental limits on the mass of an extra neutral gauge boson \(Z'\) [64,65,66,67], and the bound on the \(Z-Z'\) mixing angle in this model [68] to estimate the VEVs. Doing so, we find relations for three of the remaining four scalar potential couplings. Also, the experimental limit on the Higgs mass [64,65,66,67] is used to constrain even more some couplings.

The rest of the paper is organized as follows. In Sect. 2 we introduce the generalities of the economical \(3-3-1\) model with a \({\mathbb {Z}}_2\) symmetry that allows a scalar potential with quartic terms in a biquadratic form of the field norms. In Sect. 3, taking advantage of this property, we search for constraints on the scalar potential couplings imposing the vacuum stability conditions at tree level. Specifically, we use the method of the orbit space to simplify the application of the first and second derivative tests together with the copositivity criterion. After finding clear and useful relations for the values of some scalar potential parameters, in Sect. 4, we go further applying the positivity of the scalar masses and the experimental Higgs mass in order to constrain more scalar parameters. Finally, we present our conclusions in Sect. 5.

2 The model

In order to cancel the quantum gauge anomalies, the left-handed fermions of the economical \(3-3-1\) model have to belong to the \(\left( 1,{\mathbf {3}},\,-1/3\right) \) representation for the three lepton families and to the \(\left( {\mathbf {3}},\,{\mathbf {3}},\,1/3\right) \), \(\left( {\mathbf {3}},\,\bar{{\mathbf {3}}},\,0\right) \) representations for the quark families. More specifically,

$$\begin{aligned} \text {Leptons: } F_{aL}&=\left( \nu _{a},\,e_{a},\,N_{a}^{c}\right) _{L}^{\text {T}}\sim \left( 1,{\mathbf {3}},\,-1/3\right) , \end{aligned}$$
(1)
$$\begin{aligned} \text {Quarks: }Q_{L}&=\left( u_{1},\,d_{1},\,u{}_{4}\right) _{L}^{\text {T}}\sim \left( {\mathbf {3}},\,{\mathbf {3}},\,1/3\right) ,\nonumber \\ Q_{bL}&=\left( d_{b},\,u_{b},\,d_{b+2}\right) _{L}^{\text {T}}\sim \left( {\mathbf {3}},\,\bar{{\mathbf {3}}},\,0\right) , \end{aligned}$$
(2)

where \(a=1,2,3\), \(b=2,\,3\) and “\(\sim \)” means the transformation properties under the local symmetry group. Furthermore, in the right-handed field sector we have

$$\begin{aligned} \text {Leptons: }e_{aR}&\sim \left( 1,\,1,\,-1\right) , \end{aligned}$$
(3)
$$\begin{aligned} \text {Quarks:}\,\ u_{sR}&\sim \left( 3,\,1,\,2/3\right) ,\quad d_{tR}\sim \left( 3,\,1,\,-1/3\right) , \end{aligned}$$
(4)

where a takes the same values as in Eq. (1), \(s=1,\dots ,4\) and \(t=1,\dots ,5\).

It is also necessary to introduce, at least, three \(\hbox {SU}\left( 3\right) _{L}\) triplets, \(\rho ,\,\eta ,\,\chi \), in order to generate the appropriate fermion and boson masses. Note that the scalar sector can not be additionally reduced due to the appearance of accidental symmetries implying some massless “u” and “d” type quarks at all orders in perturbation theory as shown in Ref. [69]. In other words, the three \(\hbox {SU}\left( 3\right) _{L}\) scalar triplets are necessary to totally break the \(\text {G}_{331}\) symmetry to the \(\text {U}\left( 1\right) _{Q}\) symmetry and, at the same time, to give the phenomenologically appropriate masses for the quarks. These scalar triplets are given by

$$\begin{aligned}&\rho =\left( \rho _{1}^{+},\,\rho _{2}^{0},\,\rho _{3}^{+}\right) ^{\text {T}}\sim \left( 1,\,{\mathbf {3}},\,2/3\right) ,\nonumber \\&\eta =\left( \eta _{1}^{0},\,\eta _{2}^{-},\,\eta _{3}^{0}\right) ^{\text {T}}\sim \left( 1,\,{\mathbf {3}},\,-1/3\right) , \end{aligned}$$
(5)
$$\begin{aligned}&\chi =\left( \chi _{1}^{0},\,\chi _{2}^{-},\,\chi _{3}^{0}\right) ^{\text {T}}\sim \left( 1,\,{\mathbf {3}},\,-1/3\right) . \end{aligned}$$
(6)

Once these fermionic and bosonic fields are introduced in the model, the most general Yukawa Lagrangian, renormalizable and invariant under the local gauge group, reads

$$\begin{aligned} {\mathscr {L}}_{\text {Yuk}}={\mathscr {L}}_{\text {Yuk}}^{\rho }+{\mathscr {L}}_{\text {Yuk}}^{\eta }+{\mathscr {L}}_{\text {Yuk}}^{\chi }, \end{aligned}$$
(7)

with

$$\begin{aligned} {\mathscr {L}}_{\text {Yuk}}^{\rho }= & {} \alpha _{t}{\bar{Q}}_{L}d_{tR}\rho +\alpha _{bs}{\bar{Q}}_{bL}u_{sR}\rho ^{*}\nonumber \\&+\text {Y}_{aa^\prime }\varepsilon _{ijk}\left( {\bar{F}}_{aL}\right) _{i}\left( F_{a^\prime L}\right) _{j}^{c}\left( \rho ^{*}\right) _{k}+\text {Y}^\prime _{aa^\prime }{\bar{F}}_{aL}e_{a^\prime R}\rho \nonumber \\&+\text {H.c.,} \end{aligned}$$
(8)
$$\begin{aligned} {\mathscr {L}}_{\text {Yuk}}^{\eta }= & {} \beta _{s}{\bar{Q}}_{L}u_{sR}\eta +\beta {}_{bt}{\bar{Q}}_{bL}d_{tR}\eta ^{*}+\text {H.c.}, \end{aligned}$$
(9)
$$\begin{aligned} {\mathscr {L}}_{\text {Yuk}}^{\chi }= & {} \gamma _{s}{\bar{Q}}_{L}u_{sR}\chi +\gamma {}_{bt}{\bar{Q}}_{bL}d_{tR}\chi ^{*}+\text {H.c.}, \end{aligned}$$
(10)

where \(\varepsilon _{ijk}\) is the Levi-Civita symbol and \(a^\prime ,i,j,k=1,2,3\) and a, b, s, t are in the same range as in Eqs. (1–4). It is also straightforward to write down the most general scalar potential consistent with gauge invariance and renormalizability as

$$\begin{aligned} V\left( \eta ,\rho ,\chi \right)= & {} -\mu _{1}^{2}\eta ^{\dagger }\eta -\mu _{2}^{2}\rho ^{\dagger }\rho -\mu _{3}^{2}\chi ^{\dagger }\chi -\mu _{4}^{2}\chi ^{\dagger }\eta \nonumber \\&+\lambda _{1}\left( \eta ^{\dagger }\eta \right) ^{2}+\lambda _{2}\left( \rho ^{\dagger }\rho \right) ^{2}+\lambda _{3}\left( \chi ^{\dagger }\chi \right) ^{2} \nonumber \\&+\lambda _{4}\left( \chi ^{\dagger }\chi \right) \left( \eta ^{\dagger }\eta \right) +\lambda _{5}\left( \chi ^{\dagger }\chi \right) \left( \rho ^{\dagger }\rho \right) \nonumber \\&+\lambda _{6}\left( \eta ^{\dagger }\eta \right) \left( \rho ^{\dagger }\rho \right) +\lambda _{7}\left( \chi ^{\dagger }\eta \right) \left( \eta ^{\dagger }\chi \right) \nonumber \\&+\lambda _{8}\left( \chi ^{\dagger }\rho \right) \left( \rho ^{\dagger }\chi \right) +\lambda _{9}\left( \eta ^{\dagger }\rho \right) \left( \rho ^{\dagger }\eta \right) \nonumber \\&+\lambda _{10}\left( \chi ^{\dagger }\eta \right) ^{2}+\lambda _{11}\left( \chi ^{\dagger }\eta \right) \left( \eta ^{\dagger }\eta \right) \nonumber \\&+\lambda _{12}\left( \chi ^{\dagger }\eta \right) \left( \chi ^{\dagger }\chi \right) +\lambda _{13}\left( \chi ^{\dagger }\eta \right) \left( \rho ^{\dagger }\rho \right) \nonumber \\&+\lambda _{14}\left( \chi ^{\dagger }\rho \right) \left( \rho ^{\dagger }\eta \right) -\frac{\lambda _{15}}{\sqrt{2}}\varepsilon _{ijk}\eta _{i}\rho _{j}\chi _{k}+\text {H.c.}\,.\nonumber \\ \end{aligned}$$
(11)

From Eqs. (5) and (6), we can see that there are five electrically neutral scalars, \(\rho _{2}^{0},\) \(\eta _{1}^{0},\) \(\,\eta _{3}^{0},\) \(\chi _{1}^{0},\) \(\chi _{3}^{0}\,\) and, in principle, all of them can gain VEVs. However, it is well known that the minimal vacuum structure needed to give masses for all the particles in the model is

$$\begin{aligned}&\left\langle \rho \right\rangle =\frac{1}{\sqrt{2}}\left( 0,\,v_{\rho },\,0\right) ^{\text {T}},\quad \left\langle \eta \right\rangle =\frac{1}{\sqrt{2}}\left( v_{\eta },\,0,\,0\right) ^{\text {T}}, \nonumber \\&\left\langle \chi \right\rangle =\frac{1}{\sqrt{2}}\left( 0,\,0,\,v_{\chi }\right) ^{\text {T}}, \end{aligned}$$
(12)

which correctly reduces the \(\text {G}_{331}\) symmetry to the \(\text {U}\left( 1\right) _{Q}\) one if \(v_{\eta }^2+v_{\rho }^2\equiv v^2=246^2\) \(\hbox {GeV}^2\). There is at least another reason for choosing the minimal vacuum structure given in Eq. (12). If the remaining neutral scalars, \(\eta _{3}^{0}\) and \(\chi _{1}^{0}\), also gain VEVs, dangerous Nambu-Goldstone bosons can arise in the physical spectrum, as shown in Ref. [37]. Therefore, in order to avoid this issue and looking for simplicity, in this paper we are going to consider only the minimal vacuum structure given in Eq. (12).

We also consider the model with an extra simplifying assumption that is quite common in the \(3-3-1\) literature [37, 38, 41, 53,54,55,56,57]. It consists on the imposition of a discrete \({\mathbb {Z}}_2\) symmetry given by: \(\chi \rightarrow -\chi \), \(u_{4R}\rightarrow -u{}_{4R}\), \(\,d{}_{\left( 4,5\right) R}\rightarrow -d{}_{\left( 4,5\right) R}\) and the other fields being even under \({\mathbb {Z}}_2\). This symmetry not only brings simplicity to the model, allowing, for instance, to interpret the \(\chi \) scalar as the responsible for the first step in the symmetry breaking pattern, but also to mitigate the FCNC issues [70]. Moreover, with this \({\mathbb {Z}}_{2}\) symmetry the PQ mechanism can be easily implemented [41] and in some cases dark matter candidates can be stabilized [35, 55]. In this scenario, the Yukawa Lagrangian interactions given in Eqs. (8–10) are slightly modified to

$$\begin{aligned} {\mathscr {L}}_{\text {Yuk},\,{\mathbb {Z}}_{2}}^{\rho }= & {} \alpha _{a}{\bar{Q}}_{L}d_{aR}\rho +\alpha _{ba}{\bar{Q}}_{bL}u_{aR}\rho ^{*}\nonumber \\&+\text {Y}_{aa'}\varepsilon _{ijk}\left( {\bar{F}}_{aL}\right) _{i}\left( F_{bL}\right) _{j}^{c}\left( \rho ^{*}\right) _{k}+\text {Y}'_{aa'}{\bar{F}}_{aL}e_{a'R}\rho \nonumber \\&+ \text {H.c.,} \end{aligned}$$
(13)
$$\begin{aligned} {\mathscr {L}}_{\text {Yuk},\,{\mathbb {Z}}_{2}}^{\eta }= & {} \beta _{a}{\bar{Q}}_{L}u_{aR}\eta +\beta {}_{ba}{\bar{Q}}_{bL}d_{aR}\eta ^{*}+\text {H.c.}, \end{aligned}$$
(14)
$$\begin{aligned} {\mathscr {L}}_{\text {Yuk},\,{\mathbb {Z}}_{2}}^{\chi }= & {} \gamma _{4}{\bar{Q}}_{L}u_{4R}\chi +\gamma {}_{b\left( b+2\right) }{\bar{Q}}_{bL}d_{\left( b+2\right) R}\chi ^{*}+\text {H.c.}\,.\nonumber \\ \end{aligned}$$
(15)

Note that in this case the \(\chi \) triplet only couples to the \(u_{4R}\) and \(d_{(4,5)R}\) quarks, and it is also the reason to interpret \(\chi \) as the scalar responsible for the first step in the symmetry breaking pattern. Furthermore, after imposing the \({\mathbb {Z}}_2\) symmetry, the scalar potential becomes

$$\begin{aligned} V\left( \eta ,\rho ,\chi \right)= & {} V_{{\mathbb {Z}}_{2}}\left( \eta ,\rho ,\chi \right) + V_{\mathrm {Soft}} \left( \eta ,\rho ,\chi \right) , \nonumber \\= & {} -\mu _{1}^{2}\eta ^{\dagger }\eta -\mu _{2}^{2}\rho ^{\dagger }\rho -\mu _{3}^{2}\chi ^{\dagger }\chi \nonumber \\&+\lambda _{1}\left( \eta ^{\dagger }\eta \right) ^{2}+\lambda _{2}\left( \rho ^{\dagger }\rho \right) ^{2}+\lambda _{3}\left( \chi ^{\dagger }\chi \right) ^{2}\nonumber \\&+\lambda _{4}\left( \chi ^{\dagger }\chi \right) \left( \eta ^{\dagger }\eta \right) +\lambda _{5}\left( \chi ^{\dagger }\chi \right) \left( \rho ^{\dagger }\rho \right) \nonumber \\&+\lambda _{6}\left( \eta ^{\dagger }\eta \right) \left( \rho ^{\dagger }\rho \right) +\lambda _{7}\left( \chi ^{\dagger }\eta \right) \left( \eta ^{\dagger }\chi \right) \nonumber \\&+\lambda _{8}\left( \chi ^{\dagger }\rho \right) \left( \rho ^{\dagger }\chi \right) +\lambda _{9}\left( \eta ^{\dagger }\rho \right) \left( \rho ^{\dagger }\eta \right) \nonumber \\&+\lambda _{10}\left( \chi ^{\dagger }\eta \right) ^{2} -\frac{\lambda _{15}}{\sqrt{2}}\varepsilon _{ijk}\eta _{i}\rho _{j}\chi _{k}+\text {H.c.}\,. \end{aligned}$$
(16)

It is important to note that, even though the term \(\lambda _{15}\varepsilon _{ijk}\eta _{i}\rho _{j}\chi _{k}\) softly breaks the \({\mathbb {Z}}_2\) symmetry, it must be included because in its absence appears a QCD axion with a small decay constant, \(11.5\text { keV}\le f_{a}\le 246\text { GeV}\), already ruled out by experiments [34, 71]. See Ref. [37] for a detailed study of this case.

3 Minimization and vacuum stability

Now, we turn our attention to find constraints on the \(\mu _i-\) and \(\lambda _i-\)scalar parameters coming from minimization and vacuum stability. The general minimization of the scalar potential in Eq. (16) is a difficult task due to the large number of free parameters in the scalar potential (14 free parameters), the large number of components of the scalars triplets in the model (18 components for the \(\rho ,\,\eta ,\,\chi \) triplets) and the degeneracies of the extremal points of the potential required by the \(\text {G}_{331}-\)invariance. Fortunately, there is a powerful tool to simplify this problem which consists in working with the norm of the fields and orbit parameters. This method to minimize scalar potentials, also known as the method of the orbit space, is detailed in Refs. [60,61,62,63] in the context of spontaneous symmetry breaking. It has been used, for instance, in models with \(\text {SU}(5)\) and \(\text {SO}(10)\) gauge symmetries when scalars belong to different representations [60, 72,73,74]. The crucial observation of the method is that working with the norm of the fields, \(|\phi |\) (\(|\phi |^2 \equiv \phi ^{*}_{k}\phi _k\) - where a sum over repeated indices is implied) and the invariant orbit parameters \(\varvec{\theta }\)’s (generically defined by \({\varvec{\theta }}=\frac{f_{ijkl}\,\phi ^{*}_{i}\phi _{j}\phi ^{*}_k\phi _l}{|\phi |^4}\)) contain all the information needed to determine the minimum of the potential and, in addition, greatly reduce the number of variables.

In order to apply the method of orbit space, let’s define the invariant orbit parameters of the scalar potential in Eq. (16) as

$$\begin{aligned}&\varvec{\theta }_1({\hat{\eta }},{\hat{\chi }}) = {\hat{\chi }}^*_j{\hat{\eta }}_j{\hat{\eta }}^*_i{\hat{\chi }}_i, \quad {\varvec{\theta }}_2 ({\hat{\rho }},{\hat{\chi }}) = {\hat{\chi }}^*_j{\hat{\rho }}_j{\hat{\rho }}^*_i{\hat{\chi }}_i, \quad \nonumber \\&{\varvec{\theta }}_3 ({\hat{\eta }},{\hat{\rho }}) = {\hat{\eta }}^*_i{\hat{\rho }}_i {\hat{\rho }}^*_j{\hat{\eta }}_j,\quad \varvec{\theta }_4({\hat{\eta }},{\hat{\chi }}) = ({\hat{\chi }}^*_i {\hat{\eta }}_i)^2+ {\mathrm {H.c.}}, \quad \nonumber \\&{\varvec{\theta }}_5 ({\hat{\eta }},{\hat{\rho }},{\hat{\chi }}) = \varepsilon _{ijk}{\hat{\eta }}_i {\hat{\rho }}_j {\hat{\chi }}_k + {\mathrm {H.c.}}\, , \end{aligned}$$
(17)

where \({\hat{\phi }}_k = \phi _k / |\phi |\) and a sum over the repeated indices is implied. Note that all directional information is contained within the \(\varvec{\theta }\) parameters.

The scalar potential takes the following simple form when written using the \(\varvec{\theta }\) parameters:

$$\begin{aligned} V(\eta ,\rho ,\chi )= & {} -\mu _1^2 |\eta |^2 -\mu _2^2 | \rho |^2 -\mu _3^2 | \chi |^2 \nonumber \\&+ \lambda _1 |\eta |^4 +\lambda _2 | \rho |^4 +\lambda _3| \chi |^4 \nonumber \\&+ (\lambda _4 + \lambda _7 \varvec{\theta }_1 + |\lambda _{10}| \varvec{\theta }_4) |\eta |^2|\chi |^2\nonumber \\&+ (\lambda _5 +\lambda _8 \varvec{\theta }_2) |\rho |^2 |\chi |^2 + (\lambda _6 +\lambda _9 \varvec{\theta }_3) |\eta |^2|\rho |^2 \nonumber \\&-\frac{|\lambda _{15}|}{\sqrt{2}} \varvec{\theta }_5\, |\eta | |\rho | |\chi |, \end{aligned}$$
(18)

where we have used the fact that making a redefinition of the scalar fields, e.g. \(\eta _k \rightarrow e^{-i\delta _{10}/4} \eta _k, \,\chi _k \rightarrow e^{i\delta _{10}/4}\chi _k\) and \(\rho _k \rightarrow e^{-i\delta _{15}} \rho _k\), the phases of \(\lambda _{10}=|\lambda _{10}|\) \(e^{i\delta _{10}}\) and \(\lambda _{15} = |\lambda _{15}|\) \(e^{i\delta _{15}}\) couplings can be absorbed. Therefore, all couplings in the scalar potential can be considered as real numbers, without loss of generality.

Treating the modules of \(\eta \), \(\rho \), \(\chi \), and the \(\varvec{\theta }\) parameters as independent variables, we could apply the first and second derivative tests to provide general conditions to have a minimum of the scalar potential in Eq. (16). However, we will restrict ourselves to the phenomenologically interesting vacuum given in Eq. (12). In other words, we consider the directional minima in the particular direction given by

$$\begin{aligned} (\varvec{\theta }_1)_0 = 0,\quad (\varvec{\theta }_2)_0 = 0, \quad (\varvec{\theta }_3)_0 = 0,\quad (\varvec{\theta }_4)_0 = 0,\quad (\varvec{\theta }_5)_0 = 2.\nonumber \\ \end{aligned}$$
(19)

Thus, taking the directional derivative of the scalar potential in Eq. (18) in relation to the norm of the scalar fields, we obtain an expression for the \(\mu _i\) parameters:

$$\begin{aligned}&2\mu _1^2=2\lambda _1 v_{\eta }^2+\lambda _6 v_{\rho }^2+\lambda _4 v_{\chi }^2-|\lambda _{15}| v_{\rho } v_{\chi }/v_{\eta } , \nonumber \\&2\mu _2^2=2\lambda _2 v_{\eta }^2+\lambda _6 v_{\rho }^2+\lambda _5 v_{\chi }^2-|\lambda _{15}| v_{\eta } v_{\chi }/v_{\rho }, \nonumber \\&2\mu _3^2=2\lambda _4 v_{\eta }^2+\lambda _5 v_{\rho }^2+\lambda _3 v_{\chi }^2-|\lambda _{15}| v_{\eta } v_{\rho }/v_{\chi }, \end{aligned}$$
(20)

where we have used \(|\eta |_0=v_{\eta }/\sqrt{2} \), \(|\rho |_0=v_{\rho }/\sqrt{2} \) and \(|\chi |_0=v_{\chi }/\sqrt{2} \) which are the norm of the fields in the vacuum direction in Eq. (12).

Additional conditions on the scalar potential parameters come from second derivative test. Specifically, all principal minors of the Hessian matrix \(H_0\) evaluated at the vacuum, \((H_0)_{ij} = \frac{\partial ^2 V}{\partial \phi _i \partial \phi _j} \bigg |_{\phi =\phi _0}, \) must be positive (see for example [75, 76]). A straightforward calculation gives the Hessian matrix at the directional minimum as:

$$\begin{aligned} H_0 = \left( \begin{array}{ccc} 4\lambda _1 v_{\eta }^2+|\lambda _{15}| v_{\rho } v_{\chi }/ v_{\eta } &{}\quad 2 \lambda _6 v_{\eta } v_{\rho } -|\lambda _{15}| v_{\chi } &{}\quad 2\lambda _4 v_{\eta } v_{\chi } - |\lambda _{15}| v_{\rho } \\ \star &{}\quad 4\lambda _2 v_{\rho }^2+|\lambda _{15}| v_{\eta } v_{\chi } /v_{\rho } &{}\quad 2 \lambda _5 v_{\rho } v_{\chi } - |\lambda _{15}|v_{\eta }\\ \star &{}\quad \star &{}\quad 4\lambda _3 v_{\chi }^2+|\lambda _{15}| v_{\eta } v_{\rho } /v_{\chi } \end{array}\right) , \end{aligned}$$
(21)

where we have used the relations in Eq. (20) in order to simplify the Hessian matrix. From the positivity of the principal minors we get the conditions

$$\begin{aligned}&\lambda _1>-|\lambda _{15}| v_\rho v_{\chi }/4v_{\eta }^3 ,\,\lambda _2>-|\lambda _{15}| v_{\eta } v_{\chi }/4v_{\rho }^3,\, \lambda _3>-|\lambda _{15}| v_{\eta } v_{\rho }/4v_{\chi }^3, \nonumber \\&\quad \frac{ |\lambda _{15}| v_\rho -\sqrt{(H_0)_{11}(H_0)_{33}}}{2v_\eta v_\chi }< \lambda _4< \frac{ |\lambda _{15}| v_\rho +\sqrt{(H_0)_{11}(H_0)_{33}}}{2v_\eta v_\chi }, \nonumber \\&\quad \frac{ |\lambda _{15}|v_\eta -\sqrt{(H_0)_{22}(H_0)_{33}}}{2 v_\rho v_\chi }< \lambda _5< \frac{ |\lambda _{15}|v_\eta +\sqrt{(H_0)_{22}(H_0)_{33}}}{2 v_\rho v_\chi }, \nonumber \\&\quad \frac{|\lambda _{15}| v_\chi -\sqrt{(H_0)_{11}(H_0)_{22}}}{2 v_\eta v_\rho }< \lambda _6 < \frac{|\lambda _{15}| v_\chi +\sqrt{(H_0)_{11}(H_0)_{22}}}{2 v_\eta v_\rho }, \nonumber \\&\quad \det \, H_0 >0, \end{aligned}$$
(22)

where we have not written explicitly an analytical expression for the determinant of \(H_0\), which is not very enlightening. In addition, \(\det \, H_0>0\) will be automatically satisfied provided we consider the positivity of the square masses of the CP-even scalars as will be shown in the next section.

So far, we have studied the conditions for the vacuum configuration in Eq. (12) to be a minimum of the scalar potential. Nevertheless, for the scalar potential in Eq. (18) to make physical sense, it has to be bounded from below in the large field limit for all possible directions in the field space. To obtain these constraints it is enough to work with the quartic terms, which can be written in a matrix form as

$$\begin{aligned} V_4= & {} \lambda _1 |\eta |^4 +\lambda _2 | \rho |^4 +\lambda _3| \chi |^4 + (\lambda _4 + \lambda _7 \varvec{\theta }_1 +|\lambda _{10}| \varvec{\theta }_4) |\eta |^2|\chi |^2 \nonumber \\&+ (\lambda _5 +\lambda _8 \varvec{\theta }_2) |\rho |^2 |\chi |^2 + (\lambda _6 +\lambda _9 \varvec{\theta }_3) |\eta |^2|\rho |^2, \nonumber \\\equiv & {} \mathbf{h }^{\text {T}} \varLambda \mathbf{h }, \end{aligned}$$
(23)

where \(\mathbf{h }^{\text {T}}=(|\eta |^2,\, |\rho |^2,\, |\chi |^2)\ge 0\) and the matrix \(\varLambda \) is

$$\begin{aligned} \varLambda = \begin{pmatrix} \lambda _1 &{}\quad (\lambda _6 +\lambda _9 \varvec{\theta }_3)/2 &{}\quad (\lambda _4 + \lambda _7 \varvec{\theta }_1 + |\lambda _{10}| \varvec{\theta }_4) /2\\ \star &{}\quad \lambda _2 &{}\quad (\lambda _5 +\lambda _8 \varvec{\theta }_2)/2\\ \star &{}\quad \star &{}\quad \lambda _3 \end{pmatrix}.\nonumber \\ \end{aligned}$$
(24)

Note that \(V_4\) is a biquadratic form of the norm of the fields. Therefore, the scalar potential is bounded from below, \(\mathbf{h }^{\text {T}}\varLambda \mathbf{h } > 0\), if the symmetric matrix \(\varLambda \) is strictly copositive, i.e. positive on positive vectors \(\mathbf{h } > 0\). Note that we use the requirement of strong stability (\(V_4>0\)) because the stability in the marginal sense (\(V_4\ge 0\)) does not allow any cubic terms in the scalar potential. However, the term \(\lambda _{15}\varepsilon _{ijk}\eta _{i}\rho _{j}\chi _{k}\) must be included because in its absence appears a visible QCD axion (see Ref. [37]).

Mathematical conditions for a general symmetric matrix being strictly copositive were found in [58, 59] and applied in some elementary particle physics models in Refs. [61, 77]. For the case of a symmetric matrix \(\hbox {A}_{3\times 3}\) of special interest for this paper, these conditions read [78, 79]

$$\begin{aligned}&a_{11}>0, \quad a_{22}>0,\quad a_{33}> 0,\nonumber \\&{\bar{a}}_{12}\equiv a_{12}+\sqrt{a_{11}a_{22}}>0, \quad \nonumber \\&{\bar{a}}_{13}\equiv a_{13}+\sqrt{a_{11}a_{33}}>0, \quad {\bar{a}}_{23}\equiv a_{23}+\sqrt{a_{22}a_{33}}>0,\nonumber \\&a_{12}\sqrt{a_{33}}+a_{13}\sqrt{a_{22}}+a_{23}\sqrt{a_{11}}\nonumber \\&\quad +\sqrt{a_{11}a_{22}a_{33}}+\sqrt{2{\bar{a}}_{12}{\bar{a}}_{13}{\bar{a}}_{23}}> 0, \end{aligned}$$
(25)

where \(a_{ij}\) stands for a generic element of the \(\hbox {A}_{3\times 3}\) matrix.

In order to apply these conditions in our particular case, it is convenient to rewrite the orbit parameters involved in the definition of \(\varLambda \) as

$$\begin{aligned}&\varvec{\theta }_1= {\hat{\chi }}^*_j{\hat{\eta }}_j{\hat{\eta }}^*_i{\hat{\chi }}_i=|\varvec{\theta }_1|, \quad {\varvec{\theta }}_2 ={\hat{\chi }}^*_j{\hat{\rho }}_j{\hat{\rho }}^*_i{\hat{\chi }}_i=|\varvec{\theta }_2|, \nonumber \\&{\varvec{\theta }}_3= {\hat{\eta }}^*_i{\hat{\rho }}_i {\hat{\rho }}^*_j{\hat{\eta }}_j=|\varvec{\theta }_3|, \quad \varvec{\theta }_4 = ({\hat{\chi }}^*_i {\hat{\eta }}_i)^2+ {\mathrm {H.c.}}=2|\varvec{\theta }_1|\cos (\omega _{\varvec{\theta }_1}),\nonumber \\ \end{aligned}$$
(26)

where \(\omega _{\varvec{\theta }_1}\) is defined through \({\hat{\chi }}^*_j{\hat{\eta }}_j=\sqrt{|\varvec{\theta }_1|}\exp (\omega _{\varvec{\theta }_1}/2)\) and \(0 \le |\varvec{\theta }_{1,2,3}|\le 1\). These orbit parameters can be determined from the following observation. If \(V_4<0\) for a particular direction with \(|\mathbf{h }| =1\) determined by a set of orbit parameters, then the scalar potential will not be bounded from below in the limit \(|\mathbf{h }| \longrightarrow \infty \). Therefore, to determine whether \(V_4\) is stable in the limit of large field values, it is sufficient to find its minimum on a \(|\mathbf{h }| =1\) hypersphere. Doing so, we immediately notice that \(\omega _{\varvec{\theta }_1}=\pi \). The remaining orbit parameters are calculated by noting that \(V_4\) is a monotonic function of them. Thus, the minimum is in some \(|\varvec{\theta }_{i}|\) values (with \(i=1,2,3\)) on the frontier of their space.

At first glance, we could think the frontier for the \(|\varvec{\theta |}\) is a cube with edge length equals one. However, from the definition of \(|\varvec{\theta |}\) given in Eq. (26), it can be found, without loss of generality, that the frontier is given by

$$\begin{aligned}&0 \le |\varvec{\theta }_{1}|\le 1, \quad 0 \le |\varvec{\theta }_{2}|\le 1, \nonumber \\&\quad \max \left[ 0,\,\sqrt{|\varvec{\theta }_{1}||\varvec{\theta }_{2}|}-\sqrt{\left( 1-|\varvec{\theta }_{1}|\right) \left( 1-|\varvec{\theta }_{2}|\right) } \right] ^2 \le |\varvec{\theta }_{3}| \nonumber \\&\quad \le \left[ \sqrt{|\varvec{\theta }_{1}||\varvec{\theta }_{2}|}+\sqrt{\left( 1-|\varvec{\theta }_{1}|\right) \left( 1-|\varvec{\theta }_{2}|\right) } \right] ^2. \end{aligned}$$
(27)

In Fig. 1 we can see that the space of \(|\varvec{\theta }_{1,2,3}|\) is actually smaller than the cube. In general, note that \(|\varvec{\theta }_{1,2,3}|\) are independent orbit parameters; however, their frontiers are fixed by Eq. (27). Although we know the values of \(|\varvec{\theta }_{1,2,3}|\) which minimize \(V_4\) on the hypersphere with \(|\mathbf{h }| =1\) are in the frontier given in Fig. 1, their specific values depend on the \(\lambda _i\) coefficients in the quartic scalar potential. Among the eight general cases, there are five cases where the minimizing \(|\varvec{\theta }_{1,2,3}|\) values can be easily found. These cases are:

  1. (1)

    \(|\varvec{\theta }_{1,2,3}|=0\),    if \(\lambda _7-2|\lambda _{10}|\ge 0\), \(\lambda _8\ge 0\) and \(\lambda _9\ge 0\);

  2. (2)

    \(|\varvec{\theta }_{1,2}|=0\) and \(|\varvec{\theta }_{3}|=1\),    if \(\lambda _7-2|\lambda _{10}|\ge 0\), \(\lambda _8\ge 0\) and \(\lambda _9<0\);

  3. (3)

    \(|\varvec{\theta }_{1,3}|=0\) and \(|\varvec{\theta }_{2}|=1\),    if \(\lambda _7-2|\lambda _{10}|\ge 0\), \(\lambda _8<0\) and \(\lambda _9\ge 0\);

  4. (4)

    \(|\varvec{\theta }_{2,3}|=0\) and \(|\varvec{\theta }_{1}|=1\),    if \(\lambda _7-2|\lambda _{10}|<0\), \(\lambda _8\ge 0\) and \(\lambda _9\ge 0\);

  5. (5)

    \(|\varvec{\theta }_{1,2,3}|=1\),    if \(\lambda _7-2|\lambda _{10}|<0\), \(\lambda _8<0\) and \(\lambda _9<0\).

Now, by applying the copositivity conditions in Eq. (25) to the symmetric matrix \(\varLambda \) with the orbit parameters in the five previous cases, we find the following constraints on the \(\lambda \) parameters of the scalar potential:

$$\begin{aligned}&\lambda _1> 0,\quad \lambda _2>0,\quad \lambda _3>0,\nonumber \\&\lambda _4 +2\sqrt{\lambda _1 \lambda _3}>0, \quad \lambda _4 + \lambda _7 -2 |\lambda _{10}|+2\sqrt{\lambda _1 \lambda _3}>0, \nonumber \\&\lambda _5 + 2\sqrt{\lambda _2 \lambda _3}>0,\quad \lambda _5 +\lambda _8 + 2\sqrt{\lambda _2 \lambda _3}>0, \nonumber \\&\lambda _6 + 2\sqrt{\lambda _1 \lambda _2}>0, \quad \lambda _6 +\lambda _9 + 2\sqrt{\lambda _1 \lambda _2}>0, \end{aligned}$$
(28)

and \(2^3=8\) additional inequalities with the following form:

$$\begin{aligned} \text {C}_{1}\sqrt{\lambda _2}+\text {C}_{2}\sqrt{\lambda _1}+\text {C}_{3}\sqrt{\lambda _3} + 2\sqrt{\lambda _1 \lambda _2 \lambda _3} + \sqrt{\overline{\text {C}}_{1}\overline{\text {C}}_{2}\overline{\text {C}}_{3}} > 0,\nonumber \\ \end{aligned}$$
(29)

where \( \text {C}_1=\lbrace \lambda _4,\,\lambda _4 + \lambda _7 -2 |\lambda _{10}|\rbrace \), \( \text {C}_2=\lbrace \lambda _5,\,\lambda _5 +\lambda _8 \rbrace \), \( \text {C}_3= \lbrace \lambda _6,\,\lambda _6 +\lambda _9 \rbrace \), \(\overline{\text {C}}_1= \text {C}_1+2\sqrt{\lambda _1\lambda _{3}}\), \( \overline{\text {C}}_2=\text {C}_2+2\sqrt{\lambda _2\lambda _{3}}\) and \(\overline{\text {C}}_3=\text {C}_3+2\sqrt{\lambda _1\lambda _{2}}\). Note that these 17 inequalities are the same as those obtained if we had considered the cube with side one as the orbit parameter space.

Fig. 1
figure 1

Allowed space for the orbit parameters \(\varvec{\theta }_{1,2,3}\) defined in Eq. (26). Note that the allowed volume in this space is not a cube, but a closed volume with boundaries given in Eq. (27). Here, it is important to remark that a similar reduction of the orbit parameter space happens in the context of the triplet seesaw model [80]. In that case, the relation on the boundary between the two orbit parameters makes that the allowed space is not longer a box. Thus, the stability conditions at tree level in that model are modified to a set less restrictive, correcting what had been done in the previous literature

Full size image

The remaining three cases: (6) \(\lambda _7-2|\lambda _{10}|\ge 0\), \(\lambda _8<0\) and \(\lambda _9<0\); (7) \(\lambda _7-2|\lambda _{10}|<0\), \(\lambda _8\ge 0\) and \(\lambda _9<0\); and (8) \(\lambda _7-2|\lambda _{10}|<0\), \(\lambda _8<0\) and \(\lambda _9\ge 0\) are more complicated because the minimizing \(|\varvec{\theta }_{1,2,3}|\) parameters, \(|\varvec{\theta }_{1,2,3}|^{\star }\), depend not only on the signal of \(\lambda _7-2|\lambda _{10}|\), \(\lambda _8\) and \(\lambda _9\) but also on the values of the \(\lambda _i\) couplings . In other words, the \(|\varvec{\theta }_{1,2,3}|^{\star }\) are now functions of \(\lambda _i\) with values on the frontier in Fig. 1. In general, these cases bring inequalities that can be written as (see for example [80]): \(\quad \lambda _4 + (\lambda _7 -2 |\lambda _{10}|)|\varvec{\theta }_{1}|^{\star }+2\sqrt{\lambda _1 \lambda _3}>0\), \(\lambda _5 +\lambda _8\,|\varvec{\theta }_{2}|^{\star } + 2\sqrt{\lambda _2 \lambda _3}>0\) and \(\lambda _6 +\lambda _9\,|\varvec{\theta }_{3}|^{\star } + 2\sqrt{\lambda _1 \lambda _2}>0\). Although \(|\varvec{\theta }_{1,2,3}|^{\star }(\lambda _i)\) depend on the specific values of \(\lambda _i\) couplings, these still satisfy that \(0\le |\varvec{\theta }_{1,2,3}|^{\star }(\lambda _i)\le 1\). That is a key point, because of that, we do not really need to calculate these intricate functions. In order to clarify that point, let’s consider the particular case of \(\lambda _8<0\). As \(0\le |\varvec{\theta }_{2}|^{\star }(\lambda _i)\le 1\) and \(\lambda _8<0\), then \(0 \ge \lambda _8|\varvec{\theta }_{2}|^{\star }(\lambda _i) \ge \lambda _8\). Applying general properties of inequalities, we find \(2\sqrt{\lambda _2\lambda _3} \ge \lambda _8|\varvec{\theta }_{2}|^{\star }(\lambda _i)+2\sqrt{\lambda _2\lambda _3} \ge \lambda _8+2\sqrt{\lambda _2\lambda _3}\) or in an equivalent way,

$$\begin{aligned} -\lambda _8-2\sqrt{\lambda _2\lambda _3}\ge -\lambda _8|\varvec{\theta }_{2}|^{\star }(\lambda _i)-2\sqrt{\lambda _2\lambda _3} \ge -2\sqrt{\lambda _2\lambda _3}.\nonumber \\ \end{aligned}$$
(30)

Now, from \(\lambda _5 +\lambda _8 + 2\sqrt{\lambda _2 \lambda _3}>0\) inequality in Eq. (29), we find \(\lambda _5>-\lambda _8 - 2\sqrt{\lambda _2 \lambda _3}\ge -\lambda _8|\varvec{\theta }_{2}|^{\star }(\lambda _i)-2\sqrt{\lambda _2\lambda _3}\) where we have used Eq. (30) in the last step. Now, we realize that \(\lambda _5>\lambda _8+2\sqrt{\lambda _2 \lambda _3}\ge \lambda _8|\varvec{\theta }_{2}|^{\star }(\lambda _i)+2\sqrt{\lambda _2\lambda _3}\) implies that if we satisfy the \(\lambda _5 +\lambda _8 + 2\sqrt{\lambda _2 \lambda _3}>0\) inequality, the unknown \(\lambda _5 +\lambda _8\,|\varvec{\theta }_{2}|^{\star } +2\sqrt{\lambda _2 \lambda _3}>0\) inequality is also satisfied when \(\lambda _8<0\). We remark here that this result is independent on the particular form of \(|\varvec{\theta }_{2}|^{\star }\) provided \(0\le |\varvec{\theta }_{2}|^{\star }(\lambda _i)\le 1\) is valid. For the case of \(\lambda _8>0\), we can follow an analogue procedure to arrive to the conclusion that for this case when the \(\lambda _5 + 2\sqrt{\lambda _2 \lambda _3}>0\) inequality in Eq. (29) is satisfied, then \(\lambda _5 +\lambda _8\,|\varvec{\theta }_{2}|^{\star } +2\sqrt{\lambda _2 \lambda _3}>0\) is automatically satisfied. Therefore, whatever the sign of \(\lambda _8\) is, the unknown \(\lambda _5 +\lambda _8\,|\varvec{\theta }_{2}|^{\star } +2\sqrt{\lambda _2 \lambda _3}>0\) is not necessary. Similar conclusions are obtained for the remaining two unknown inequalities: \(\quad \lambda _4 + (\lambda _7 -2 |\lambda _{10}|)|\varvec{\theta }_{1}|^{\star }+2\sqrt{\lambda _1 \lambda _3}>0\) and \(\lambda _6 +\lambda _9\,|\varvec{\theta }_{3}|^{\star } + 2\sqrt{\lambda _1 \lambda _2}>0\). Therefore, we have only 17 necessary and sufficient inequalities given by Eqs. (28) and (29) coming from the strict copositivity of \(\varLambda \), which additionally delimit the scalar potential parameters.

At first glance, obtaining analytical results from (28) and (29) seems a very complicated task. Nevertheless, there are general relations between some \(\lambda \)’s that we can find. First, we can answer, for instance, the question: which is the largest excluded region in the \(\lambda _4-\lambda _7\) plane provided the rest of the \(\lambda \) couplings satisfy the copositivity conditions? That excluded region depends on the value of \(\lambda _7\). On the one hand, if it is bigger than or equal to \(2|\lambda _{10}|\), all \(\lambda _4\) values smaller than or equal to \(2\sqrt{\lambda _1\lambda _3}\) are excluded. On the other hand, for values of \(\lambda _7\) smaller than \(2|\lambda _{10}|\), the region \(\lambda _4\le 2\sqrt{\lambda _1 \lambda _3}+2|\lambda _{10}|-\lambda _{7}\) is rejected. In both cases of \(\lambda _7\), the largest excluded region occur whenever \(\lambda _5=\lbrace -2\sqrt{\lambda _2\lambda _3},-2\sqrt{\lambda _2\lambda _3}-\lambda _8\rbrace \) and \(\lambda _6=\lbrace -2\sqrt{\lambda _1\lambda _2},-2\sqrt{\lambda _1\lambda _2}-\lambda _9\rbrace \) where the first value of \(\lambda _5\ (\lambda _6)\) corresponds to \(\lambda _8\ge 0\ (\lambda _9\ge 0)\) and the second one corresponds to \(\lambda _8<0(\lambda _9<0)\), independently of the other values of \(\lambda \) couplings. Once \(\lambda _5\) and \(\lambda _6\) take different values from the aforementioned ones, the excluded region for \(\lambda _4\) decreases. However, that region does not decrease forever since there is also a lower bound depending on \(\lambda _7\) coupling. In more detail, if \(\lambda _7\) is bigger than or equal to \(2|\lambda _{10}|\), all \(\lambda _4\) values smaller than or equal to \(-2\sqrt{\lambda _1\lambda _3}\) are always excluded. For the case of \(\lambda _7\) values smaller than or equal to \(2|\lambda _{10}|\), \(\lambda _4\) values smaller than or equal to \(-2\sqrt{\lambda _1\lambda _3}+2|\lambda _{10}|-\lambda _7\) are always excluded. For both cases, points satisfying the following conditions are on the bound:

  1. (1)

    \(\lambda _5\sqrt{\lambda _1}+\lambda _6\sqrt{\lambda _3}>0\),    if \(\lambda _8\ge 0\) and \(\lambda _9\ge 0\).

  2. (2)

    \(\lambda _5\sqrt{\lambda _1}+(\lambda _6+\lambda _9)\sqrt{\lambda _3}>0\),   if \(\lambda _8\ge 0\) and \(\lambda _9< 0\).

  3. (3)

    \((\lambda _5+\lambda _8)\sqrt{\lambda _1}+\lambda _6\sqrt{\lambda _3}>0\),   if \(\lambda _8<0\) and \(\lambda _9\ge 0\).

  4. (4)

    \((\lambda _5+\lambda _8)\sqrt{\lambda _1}+(\lambda _6+\lambda _9)\sqrt{\lambda _3}>0\),   if \(\lambda _8<0\) and \(\lambda _9<0\).

The maximum and minimum excluded regions for \(\lambda _4\) as a function of \(\lambda _7\) are shown in Fig. 2. Similar conclusions can be reached for the \(\lambda _5-\lambda _8\) and \(\lambda _6-\lambda _9\) planes and are shown in Fig. 3. In the left panel of Fig. 3, the maximum and the minimum excluded regions for \(\lambda _5\) as a function of \(\lambda _8\) are shown. In general, the maximum excluded region of \(\lambda _5\) depends on the sign of \(\lambda _8\), i.e. if it is negative, then \(\lambda _5\) values satisfying \(\lambda _5\le 2\sqrt{\lambda _2\lambda _3}-\lambda _8\) are excluded. Otherwise, \(\lambda _5\) values smaller than or equal to \(2\sqrt{\lambda _2\lambda _3}\) are excluded. The maximum excluded region of \(\lambda _5\) is reached when \(\lambda _4 + \left( \lambda _7 -2 |\lambda _{10}|\right) \theta _\text {H}\left( 2|\lambda _{10}| -\lambda _7\right) +2\sqrt{\lambda _1\lambda _{3}}=0\) and \(\lambda _6 +\lambda _9\,\theta _\text {H}\left( -\lambda _9\right) +2\sqrt{\lambda _1\lambda _{2}}=0\), where \(\theta _\text {H}\) is the Heaviside theta function. The minimum excluded region of \(\lambda _5\) has the same dependence on \(\lambda _8\) as the maximum excluded one, however, its bounds are different. If \(\lambda _8\) is smaller than zero, then \(\lambda _5\) values satisfying \(\lambda _5\le -2\sqrt{\lambda _2\lambda _3}-\lambda _8\) are excluded. Otherwise, \(\lambda _5\le -2\sqrt{\lambda _2\lambda _3}\) are excluded. These bounds are attained when \(\left( \lambda _4 + \left( \lambda _7 -2 |\lambda _{10}|\right) \theta _\text {H}\left( 2|\lambda _{10}| -\lambda _7\right) +2\sqrt{\lambda _1\lambda _{3}}\right) \,\sqrt{\lambda _2}+\left( \lambda _6 +\lambda _9\,\theta _\text {H}\left( -\lambda _9\right) +2\sqrt{\lambda _1\lambda _{2}}\right) \,\sqrt{\lambda _3}>4\sqrt{\lambda _1\lambda _2\lambda _3}\).

Fig. 2
figure 2

The maximum excluded region for \(\lambda _4\) is defined by \(\lambda _4\le 2\sqrt{\lambda _1\lambda _3}\) for \(\lambda _7\ge 2|\lambda _{10}|\), and \(\lambda _4\le 2\sqrt{\lambda _1\lambda _3}+2|\lambda _{10}|-\lambda _7\) for \(\lambda _7<2|\lambda _{10}|\), whereas the minimum excluded region is defined by \(\lambda _4\le -2\sqrt{\lambda _1\lambda _3}\) for \(\lambda _7\ge 2|\lambda _{10}|\), and \(\lambda _4\le -2\sqrt{\lambda _1\lambda _3}+2|\lambda _{10}|-\lambda _7\) for \(\lambda _7<2|\lambda _{10}|\)

Full size image
Fig. 3
figure 3

Maximum and minimum excluded regions in the \(\lambda _5-\lambda _8\) (left) and \(\lambda _6-\lambda _9\) (right) parameter space. In general, the maximum and minimum excluded regions for \(\lambda _5\) (\(\lambda _6\)) depend on the sign of \(\lambda _8\) (\(\lambda _9\)) as explained in the text

Full size image

On the other hand, in the right panel of Fig. 3, the maximum and the minimum excluded regions for \(\lambda _6\) as a function of \(\lambda _9\) are shown. The maximum excluded region of \(\lambda _6\) is characterized by two different bounds. \(\lambda _6\le 2\sqrt{\lambda _1\lambda _2}-\lambda _9\) are excluded if \(\lambda _9<0\) and \(\lambda _6\) values satisfying \(\lambda _6\le 2\sqrt{\lambda _1\lambda _2}\) are excluded if \(\lambda _9\ge 0\). That region is reached when \(\lambda _4 + \left( \lambda _7 -2 |\lambda _{10}|\right) \theta _\text {H}\left( 2|\lambda _{10}| -\lambda _7\right) \) \(+2\sqrt{\lambda _1\lambda _{3}}=0\) and \(\lambda _5 +\lambda _8\,\theta _\text {H}\left( -\lambda _8\right) +2\sqrt{\lambda _2\lambda _{3}}=0\). Finally, the minimum excluded region also has two different bounds. \(\lambda _6\le -2\sqrt{\lambda _1\lambda _2}-\lambda _9\) are excluded if \(\lambda _9<0\) and \(\lambda _6\) values satisfying \(\lambda _6\le -2\sqrt{\lambda _1\lambda _2}\) are excluded if \(\lambda _9\ge 0\). That region is reached when \(\left( \lambda _4 + \left( \lambda _7 -2 |\lambda _{10}|\right) \theta _\text {H}\left( 2|\lambda _{10}| -\lambda _7\right) +2\sqrt{\lambda _1\lambda _{3}}\right) \sqrt{\lambda _2}\) \(+\left( \lambda _5 +\lambda _8\,\theta _\text {H}\left( -\lambda _8\right) \right. \left. +2\sqrt{\lambda _2\lambda _{3}}\right) \sqrt{\lambda _1} >4\sqrt{\lambda _1\lambda _2\lambda _3}\).

It is also interesting to combine the copositivity conditions in Eqs. (28) and (29) with those coming from the positivity of the Hessian matrix in Eq. (22). Doing that, it is straightforward to see that the three conditions in the first line in Eq. (22) are always satisfied provided \(\lambda _1> 0,\, \lambda _2>0,\, \lambda _3 > 0\) as required by copositivity. In addition, the lower bounds on \(\lambda _{4,5,6}\) coming from Eq. (22) are always smaller than the obtained from copositivity. More specifically, the lower bound on \(\lambda _{4}\), for instance, coming from the Hessian matrix can be written as \(\lambda _4>\,-2\sqrt{\lambda _1 \lambda _3} - \frac{|\lambda _{15}|}{4v_\rho }\left( \frac{v_\rho ^2}{v_\eta v_\chi } \right) \left( \sqrt{\frac{\lambda _3}{\lambda _1}} \frac{v_\chi ^2}{v_\eta ^2} + \sqrt{\frac{\lambda _1}{\lambda _3}} \frac{v_\eta ^2}{v_\chi ^2} -2 \right) \), where we have assumed that \(|\lambda _{15}| \ll v_\eta ,v_\rho \). As \(\left( \sqrt{\frac{\lambda _3}{\lambda _1}} \frac{v_\chi ^2}{v_\eta ^2} + \sqrt{\frac{\lambda _1}{\lambda _3}} \frac{v_\eta ^2}{v_\chi ^2} -2 \right) \) is always positive, we conclude that the lower bound obtained from copositivity is stronger than the Hessian one. It is also important to note that the upper limit on \(\lambda _{4,5,6}\) coming from the positivity of the Hessian matrix becomes stronger when \(|\lambda _{15}| \ll v_\eta ,v_\rho \). For instance, in the limit \(|\lambda _{15}|\rightarrow 0\), these limits go to \(\lambda _4<2\sqrt{\lambda _1\lambda _3}\), \(\lambda _5<2\sqrt{\lambda _2\lambda _3}\) and \(\lambda _6<2\sqrt{\lambda _1\lambda _2}\) which are the bounds on \(\lambda _{4,5,6}\) of the maximum excluded region discussed previously. For more details on the behaviour of the upper and lower bounds of \(\lambda _4\) as a function of \(|\lambda _{15}|\) see Fig. 4. Similar conclusions are obtained for \(\lambda _5\) and \(\lambda _6\) couplings. It is important to note that since the term \(\frac{\lambda _{15}}{\sqrt{2}}\varepsilon _{ijk}\eta _{i}\rho _{j}\chi _{k}\) softly breaks the \({\mathbb {Z}}_2\) symmetry, it is technically natural that \(\lambda _{15}\) gets a small value in comparison with other energy levels in the model, \(v=\sqrt{v_\eta ^2+v_\rho ^2}\) and \(v_\chi \), since making \(\lambda _{15}\rightarrow 0\) increases the symmetry of the total Lagrangian.

Fig. 4
figure 4

The excluded region for \(\lambda _4\) coming from Hessian matrix criterion increases when \(|\lambda _{15}|\) decreases, and it also approaches to excluded region given by copositiviy. Similar conclusions can be reached for \(\lambda _5\) and \(\lambda _6\) couplings

Full size image

On the other hand, copositivity does not impose upper limits on \(\lambda _{4,5,6}\) and therefore, limits from Eq. (22) still apply, i.e.

$$\begin{aligned} \text {bounds from copositivity}<&\lambda _4&< \frac{ |\lambda _{15}| v_\rho +\sqrt{(H_0)_{11}(H_0)_{33}}}{2v_\eta v_\chi }, \nonumber \\ \text {bounds from copositivity}<&\lambda _5&< \frac{ |\lambda _{15}|v_\eta +\sqrt{(H_0)_{22}(H_0)_{33}}}{2 v_\rho v_\chi } ,\nonumber \\ \text {bounds from copositivity}<&\lambda _6&< \frac{|\lambda _{15}| v_\chi +\sqrt{(H_0)_{11}(H_0)_{22}}}{2 v_\eta v_\rho },\nonumber \\ \end{aligned}$$
(31)

where \((H_0)_{ii}\) with \(i=1,2,3\) is the corresponding element of the Hessian matrix. In order to conclude this section, we can say that out of fourteen initial parameters in the scalar potential in Eq. (16), we can clearly constrain ten of them by applying the copositivity criterion in addition to the first and second derivative tests. Specifically, the \(\mu _{1,2,3}\) parameters in Eq. (20), the \(0\le \lambda _{1,2,3,10}\le 4\pi \) and \(\lambda _{4,5,6}\) in Eq. (31) are explicitly constrained. Finally, needless to say that apart from the \(\lambda _{15}\) (which has dimension of mass), the rest of the \(\lambda \) parameters are constrained by perturbativity to be \(|\lambda |\le 4\pi \).

4 Scalar mass spectrum

A complementary set of constraints on the \(\lambda \)’s comes from the positivity of the square masses of the scalars of this model. To be more specific, the physical spectrum of scalars consists of four charged scalars, \(H^{\pm }\) and \(H_{V}^{\pm },\) one complex neutral field, \(H_U\), one CP-odd scalar, \(H_0\), and three CP-even scalars, h, \(H_2\) and \(H_3\), from which we define h as the SM Higgs boson. Their square masses can be written as:

$$\begin{aligned}&M_{H_0}^2=\frac{|\lambda _{15}|}{2v_{\eta }v_{\rho }v_{\chi }}\left( v_{\eta }^2v_{\rho }^2+\left( v_{\eta }^2+v_{\rho }^2\right) v_{\chi }^2\right) , \nonumber \\&M_{H_U}^2=\frac{v_{\eta }^2+v_{\chi }^2}{2v_{\eta }v_{\chi }}\left( |\lambda _{15}|v_{\rho }+\lambda _7v_{\eta }v_{\chi }\right) , \end{aligned}$$
(32)
$$\begin{aligned}&M_{H_{V}^{\pm }}^2=\frac{v_{\rho }^2+v_{\chi }^2}{2v_{\rho }v_{\chi }}\left( |\lambda _{15}|v_{\eta }+\lambda _8v_{\rho }v_{\chi }\right) , \nonumber \\&M_{H^{\pm }}^2=\frac{v^2}{2v_{\eta }v_{\rho }}\left( |\lambda _{15}|v_{\chi }+\lambda _9v_{\eta }v_{\rho }\right) . \end{aligned}$$
(33)

At this point, it is important to recall that \(\lambda _{15}\) has dimension of mass. We have not yet included the CP-even scalars in Eqs. (32–33) because their masses are more intricate and require a detailed analysis, as we show below.

From Eq. (32) we note that \( M_{H_0}^2\) is always larger than or equal to zero. However, the positivity of \(M_{H_U}^2\) requires that

$$\begin{aligned} \lambda _7 \ge -\frac{v_{\rho }}{v_{\eta }v_{\chi }}|\lambda _{15}|. \end{aligned}$$
(34)

For the case of charged scalars a stronger limit can be applied because their square masses not only have to be positive but also \(\gtrsim 80^2\,\hbox {GeV}^2\) (\(95\%\) C.L.) [67]. Applying this constraint to \(M_{H^{\pm }}^2\) and \(M_{H_{V}^{\pm }}^2\), we obtain

$$\begin{aligned}&\lambda _{8}\ge \frac{2\times 80^2}{(v_{\rho }/\text {GeV})^2+(v_{\chi }/\text {GeV})^2}-\frac{v_{\eta }}{v_{\rho }v_{\chi }}|\lambda _{15}|,\quad \text {and} \quad \nonumber \\&\lambda _{9}\ge 2\times \frac{80^2}{246^2}-\frac{v_{\chi }}{v_{\eta }v_{\rho }}|\lambda _{15}|. \end{aligned}$$
(35)

In order to estimate the lower limits for \(\lambda _7, \, \lambda _8\) and \(\lambda _9\) we bring in another piece of information. In general, in models with an extra neutral gauge boson, \(Z_2\), the mixing angle, \(\phi \), between this and the neutral gauge boson of the SM, \(Z_1\), has to be very small. For this \(3-3-1\) model, lower and upper bounds for this quantity have been found through the analysis of the \(Z_1\) invisible- and charged lepton- partial decay width plus theoretical consistency of the \(3-3-1\) models regarding the number of lepton families to be exactly 3. As a result, \(-3.98\times 10 ^{-3}\lesssim \tan \phi \lesssim 1.31\times 10^{-4}\) at \(90\%\) C.L. [68], independently of \(M_{Z_2}\) and the hadronic sector. Other studies regarding FCNC suppression and parity violation in the Cesium atom have shown similar results [81, 82]. As \(\tan \phi \) depends on \(v_{\eta }\) and \(v_{\chi }\), we can estimate their values in order to satisfy this limit. To do so, let’s write \(\tan ^2\phi \) as [83]

$$\begin{aligned} \tan ^2\phi =\frac{M_Z^2-M_{Z_1}^2}{M_{Z_2}^2-M_Z^2}, \end{aligned}$$
(36)

where the masses of the neutral gauge bosons \(Z_1\) and \(Z_2\) read

$$\begin{aligned} M_{Z_1}^2,\,M_{Z_2}^2 =\frac{1}{2}\left[ M_Z^2+M_{Z'}^2\mp \sqrt{\left( M_Z^2-M_{Z'}^2\right) ^2 +\left( 2M_{ZZ'}^2\right) ^2}\right] ,\nonumber \\ \end{aligned}$$
(37)

with

$$\begin{aligned}&M_Z^2=\frac{g^2}{4\cos ^2\theta _W}\,v^2,\nonumber \\&M_{Z'}^2=\frac{g^2}{12\left( 1-\frac{1}{3}\tan ^2\theta _W\right) }\nonumber \\&\quad \left[ (1+\tan ^2\theta _W)^2 v^2-4\tan ^2\theta _W v_{\eta }^2+4v_{\chi }^2\right] ,\nonumber \\&M_{ZZ'}^2=-\frac{g^2}{4\cos ^2\theta _W}\frac{v^2-2\cos ^2 \theta _W v_{\eta }^2}{\sqrt{3-4\sin ^2\theta _W}}. \end{aligned}$$
(38)

where \(\theta _W\) is the Weinberg’s angle and g is the coupling constant of the \(\hbox {SU}(2)_L\) group. Using \(v=246\) GeV, \(g=0.65\), \(\sin ^2 \theta _W\simeq 0.22 \), \(M_{Z_1}\simeq 91.19\) GeV [67, 84] and imposing \(-3.98\times 10 ^{-3}\lesssim \tan \phi \lesssim 1.31\times 10^{-4}\) we find the allowed region in the \(v_\chi -v_\eta \) plane. As we can see in Fig. 5, \(v_\chi \gtrsim 100\) TeV is needed to obtain significant deviations from the zero-mixing value, \(v_\eta \simeq 197.34\) GeV.

Fig. 5
figure 5

The light green region contains the values for \(v_\eta \) and \(v_\chi \) allowing \(-3.98\times 10 ^{-3}\lesssim \tan \phi \lesssim 1.31\times 10^{-4}\). \(v_\eta \) values at least \(1\%\) different from 197.34 GeV require \(v_\chi \gtrsim 100\) TeV

Full size image

From the phenomenological point of view, \(v_\chi \gtrsim 100 \text { TeV}\) is not interesting due to the fact that all new particles in the model have masses of the order of \(v_\chi \) and therefore would be unattainable by the current particle colliders. Thus, we use \(v_\eta \simeq 197.34\) GeV and \(v_\rho = \sqrt{v^2 - v^2_{\eta }} \simeq 146.88\) GeV. Now, it is also possible to estimate \(v_\chi \) from the fact that the mass of an extra neutral gauge boson, \(Z_2\), has to be larger than 2.9 TeV [67, 85]. Applying this bound in Eq. (37) and using the values for \(v_\eta \) and \(v_\rho \) obtained above, we have that \(v_\chi \gtrsim 7.31\) TeV. Thus, we can use these VEVs to estimate lower limits for \(\lambda _7, \, \lambda _8\) and \(\lambda _9\) in terms of the value of \(|\lambda _{15}|\)

$$\begin{aligned}&\lambda _7\gtrsim -1.02 \, \frac{|\lambda _{15}|}{\text {GeV}}, \quad \lambda _{8}\gtrsim 0.02-1.8\times 10^{-4}\,\frac{|\lambda _{15}|}{\text {GeV}}, \nonumber \\&\lambda _{9}\gtrsim 0.21-0.25\,\frac{|\lambda _{15}|}{\text {GeV}}, \end{aligned}$$
(39)

where we have used \(v_\chi = 7.31\) TeV. In the case that \(\lambda _{15}\ll 1\) GeV, for instance, we have \(\lambda _7\gtrsim 0, \, \lambda _8\gtrsim 0.02\) and \(\lambda _9\gtrsim 0.21\).

Fig. 6
figure 6

Constraints for the parameter space coming from the mass of the Higgs boson produces ellipses and hyperbolas in the \(\lambda _5-\lambda _4\) plane according to Eq. (42). In this particular example, we took \(\lambda _1=4+\lambda _2\), \(\lambda _3=3\), the usual values for the VEVs and \(m_h=125.18\) GeV. Notice how the conics rotate counter clockwise as \(\lambda _6\) increases, c.f. Eq. (43)

Full size image

Let’s return to the square masses of the CP-even scalars, h, \(H_2\) and \(H_3\). Their square masses are obtained from the eigenvalues of the matrix \(m_{ij} = \frac{1}{2}\frac{\partial ^2 V}{\partial \phi _i \partial \phi _j} \bigg |_{\phi =\text {min}},\) where \(\phi _i\), \(i=1,2,3\), are the real parts of the fields \(\eta _1^0\), \(\rho _2^0\) and \(\chi _3^0\), respectively. In that basis, \(m_{ij}\) coincides with \(\frac{1}{2}H_0\), where \(H_0\) is the Hessian matrix given in Eq. (21). A perturbative analysis in powers of \(v/v_\chi \) shows that \(m^2_h\propto v^2\) and \(m^2_{H_{2,3}}\propto v_\chi ^2\). For this reason, we have that \(m^2_{H_{2,3}}\gg m^2_h\) since \(v_\chi ^2\gg v^2\). This observation allows us to calculate an analytical expression for \(m^2_h\). In order to do so, it is useful to write the characteristic polynomial of \(\frac{1}{2}H_0\), P, as

$$\begin{aligned} P= & {} \det \left[ m^2\,\mathbf{{1}}_{3\times 3}-H_0/2\right] =\cdots +{\mathscr {C}}\,m^2+\cdots ,\nonumber \\= & {} m^6-{{\,\mathrm{Tr}\,}}\left[ H_0/2\right] \,m^4\nonumber \\&+\det \left[ H_0/2\right] \left( 1/m^2_h+1/m^2_{H_{2}}+1/m^2_{H_{3}}\right) \,m^2 \nonumber \\&-\det \left[ H_0/2\right] , \end{aligned}$$
(40)

where \(\mathbf{{1}}_{3\times 3}\) is the \(3\times 3\) identity matrix and \({\mathscr {C}}\) is the \(m^2\) coefficient of P when calculated from \(\det \left[ m^2\,\mathbf{{1}}_{3\times 3}-H_0/2\right] \). Now, since \(m^2_{H_{2,3}}\gg m^2_h\), we can write

$$\begin{aligned} m^2_h\simeq \frac{\det \left[ H_0/2\right] }{{\mathscr {C}}}, \end{aligned}$$
(41)

The formula for \(m^2_h\) in Eq. (41) is at \(2\%\) accuracy. Because \(m_h\) is the Higgs boson mass, it must be \(m_h=125.18\pm 0.16\) GeV [64,65,66,67]. This imposes strong constraints on \(\lambda _1,\cdots ,\lambda _6\) and \(|\lambda _{15}|\). For instance, the \(\lambda _4\) and \(\lambda _5\) values required to get the Higgs boson mass lie on ellipses or hyperbolas. The particular form of the conic appearing in the \(\lambda _5-\lambda _4\) plane depends on the other \(\lambda \) values given by Eq. (41). To be more specific, let’s write the eigenvalue equation for \(m_h^2\)

$$\begin{aligned}&\left( \lambda _2-\frac{m_h^2}{2 v_\rho ^2}\right) \lambda _4^2+\left( \lambda _1-\frac{m_h^2}{2 v_\eta ^2}\right) \lambda _5^2-\lambda _6\,\lambda _4\,\lambda _5 \nonumber \\&\quad +\lambda _3\left( \lambda _6^2-4\lambda _1\lambda _2\right) +m_h^2\left( 2\lambda _3\left( \lambda _2/v_\eta ^2+\lambda _1/v_\rho ^2\right) \right. \nonumber \\&\quad \left. +(4\lambda _1\lambda _2-\lambda _6^2)/2v_\chi ^2\right) =0, \end{aligned}$$
(42)

where we have used \(|\lambda _{15}|\ll v_\eta ,\,v_\rho \) for simplicity. Eq. (42) represents an ellipse or a hyperbola centered at the origin and rotated by an angle \(\theta \) given by

$$\begin{aligned} \tan 2\theta =\frac{\lambda _6}{\lambda _1-\lambda _2+\frac{1}{2} m_h^2\left( \frac{1}{v_\rho ^2}-\frac{1}{v_\eta ^2}\right) }. \end{aligned}$$
(43)

Other characteristics such as the size of the conic are dominated by \(\lambda _{1,2,3,6}\) and the ratios \(m_h^2/v_\rho ^2\) and \(m_h^2/v_\eta ^2\). Some particular cases are shown in Fig. 6 and the region separating ellipses and hyperbolas in the plane \(\lambda _2 - \lambda _{1}\) is shown in Fig.  7.

Fig. 7
figure 7

The region separating ellipses and hyperbolas for a \(\lambda _3\) fixed and \(\lambda _6\) varying

Full size image

Similar conclusions are obtained for the planes \(\lambda _6 - \lambda _{4}\) and \(\lambda _6 - \lambda _{5}\). Now, for the most general case, i.e. when \(|\lambda _{15}|\) is not negligible, Eq. (42) takes the most general form \(a\,\lambda _4^2+ 2 b\,\lambda _4\,\lambda _5 + c\,\lambda _5^2+2d\,\lambda _4+2f\,\lambda _5+g = 0\) where the coefficients \(a,\cdots ,g\) depend on \(\lambda _{1,2,3,6,15}\). Thus, for the general case the ellipse/hyperbola is not centered at the origin and the rotation angle also acquires a dependence on \(|\lambda _{15}|\). We are not going to consider that case in detail.

Finally, we can calculate \(m^2_{H_{2,3}}\) as a function of \(m^2_h\) as follows

$$\begin{aligned} m^2_{H_{2,3}}= & {} \frac{1}{2}{{\,\mathrm{Tr}\,}}\left[ H_0/2\right] - \frac{1}{2}m^2_h \nonumber \\&\mp \, \frac{1}{2}\sqrt{\left( {{\,\mathrm{Tr}\,}}\left[ H_0/2\right] - m^2_h \right) \left( {{\,\mathrm{Tr}\,}}\left[ H_0/2\right] + 3 m^2_h \right) + 4 {\mathscr {C}} }.\nonumber \\ \end{aligned}$$
(44)

Because we are working in the approximation of \(m_{H_{2,3}}>m_h\) and \(m_h=125.18\pm 0.16\) GeV [64,65,66,67], the positivity of \(m^2_{H_{2,3}}\) does not bring any additional constraint on the \(\lambda \)’s. Also, note that the positivity of the masses of CP-even scalars implies that \(\det \, H_0>0\) as required by the last condition in Eq. (22).

5 Conclusions

In this work, we find tree level constraints on the scalar potential couplings of the economical \(3-3-1\) model when considerations of vacuum stability and positivity of the squared scalar masses are taken into account. In particular, we consider the model with a discrete \({\mathbb {Z}}_2\) symmetry acting on \(\chi \), \(u_{4R}\), \(\,d{}_{\left( 4,5\right) R}\) fields in a non trivial way. Besides all the appealing features discussed in Sect. 2, this discrete symmetry makes the quartic terms in the scalar potential to have a biquadratic form of the norm of the fields. This allows us to apply copositivity criterion in order to guarantee that the scalar potential is bounded from below. When copositivity criterion is imposed in combination with the first and second derivative tests for the vacuum expectation values given in Eq. (12), ten of the scalar couplings are constrained. In more detail, \(\lambda _{1,2,3}\) need to be positive and the \(\mu _{1,2,3}\) parameters are completely determined, c.f. Eq. (20). Besides that, \(\lambda _{4,5,6}\) couplings are constrained from below by the copositivity and from above by the positivity of the principal minors of the Hessian matrix, Eq. (31). More interestingly, there is always an excluded region for \(\lambda _{4,5,6}\) which we called the minimum excluded region in Figs. 2 and 3, respectively. This region comes from the copositivity criterion and gives a lower bound for \(\lambda _{4,5,6}\). It is remarkable that the excluded region for these \(\lambda \) couplings also has a maximum provided all copositivity conditions are satisfied. \(\lambda _{7,8,9,10}\) play an important role in determining the form of both the minimum and the maximum excluded regions for \(\lambda _{4,5,6}\). On the other hand, copositivity does not have anything to say about the upper bound on \(\lambda _{4,5,6}\) and it is here that second derivative test is important. We analyse the role that \(|\lambda _{15}|\) (where we have applied a phase shift in the fields to make \(\lambda _{15}\) real and positive without loss of generality) has in determining that upper bound. As the \(|\lambda _{15}|\) coupling is technically small, we have studied the bound when \(|\lambda _{15}|\ll v_\eta ,v_\rho \) showing that the smallest upper bound on \(\lambda _{4,5,6}\) is always larger than \(2\sqrt{\lambda _1\lambda _3}\), \(2\sqrt{\lambda _2\lambda _3}\), \(2\sqrt{\lambda _1\lambda _2}\), respectively.

In order to constrain the rest of \(\lambda \) couplings, we turn our attention on positivity of the squared scalar masses. After finding general expressions for the masses of the charged and CP-odd scalars, we find constraints on \(\lambda _{7,8,9}\) given in Eqs. (34–35), respectively. Actually, we apply a stronger limit for the case of the masses of charged scalars since these, roughly speaking, must be heavier than 80 GeV. As the constraints on \(\lambda _{7,8,9}\) strongly depend on the VEVs, even in the case of \(|\lambda _{15}|\rightarrow 0\), we estimate the lower bounds on \(\lambda _{7,8,9}\) using the VEVs that satisfy the upper bound on the mixing angle between the two neutral gauge bosons in the model and the lower bound on the mass of \(Z'\) gauge boson. Doing that, we obtain the lower bounds in Eq. (39).

Moreover, we find approximate formulas for the squared masses of the CP-even scalars in the model. If the \(v_\rho ,v_\eta \ll v_\chi \) hierarchy is satisfied (as assumed in this \(3-3-1\) model), the squared masses of the CP-even scalars different from the Higgs boson are proportional to \(v_\chi ^2\), which allows us to find a \(2\%\) accurate formula for the Higgs squared mass, c.f. Eq. (41). Using the fact that the Higgs mass must be \(m_h=125.18\pm 0.16\) GeV [64,65,66,67], we find that \(\lambda _5-\lambda _4\), \(\lambda _6-\lambda _4\) and \(\lambda _6-\lambda _5\) satisfy the ellipse or the hyperbola general equations with coefficients determined by \(\lambda _{1,2,3,6,15}\) couplings. We outline the behaviour of such conics for the case \(|\lambda _{15}|\ll v_\eta ,v_\rho \) in Fig. 6. In that case, they are centered at the origin and their rotation angle strongly depends on \(\lambda _6\). Furthermore, we find equations for squared masses of the other two CP-even scalars in terms of \(m_h^2\) and the trace and determinant of the Hessian matrix, see Eq. (44). Because these squared masses are larger than \(m_h^2\) in our approach, their positivity do not bring new constraints on \(\lambda \) couplings.

Although the objective of this paper is to derive tree level conditions for the quartic couplings of the scalar potential coming from vacuum stability, the minimization of the scalar potential, the positivity of the squared masses of the extra scalars, the Higgs boson mass, the \({\hbox {Z}}^\prime \) gauge boson mass and its mixing angle with the SM Z boson in order to restrict the parameter space, it is also interesting to comment some modifications coming from the running of the coupling constants. Roughly speaking, we expect that major differences when compared with the SM are due to the presence of new particles, such as heavy quarks and leptons. For example, there are three additional quarks (an up-type quark and two down-type quarks) which are assumed to have masses in the scale of the \(\text {SU}(3)_L\) symmetry breaking, i.e. \(1-10\) TeV. For this reason, we expect that these quarks modify significantly the beta functions of the quartic couplings that involve the \(\chi \) scalar triplet because the \({\mathbb {Z}}_2\) symmetry acting on \(\chi \), \(u_{4R}\) and \(d_{4,5R}\) fields makes these new quarks gain masses mainly through the \(\chi \) scalar triplet. Thus, the beta functions of \(\lambda _{4,5}\) are expected to receive large contributions coming from the diagrams where the new quarks are running. However, other \(\lambda \) couplings can receive some contributions from these quarks due to the mixings between scalar mass eigenstates. Moreover, we expect that the beta function of the \(\lambda \) quartic couplings receive several positive contributions from the one-loop diagrams with scalars running in them. These positive contributions, roughly speaking, will reduce the allowed regions of the \(\lambda \) couplings in a similar way as in the triplet and inverse seesaw models [80, 86]. Nevertheless, in order to give a quantitative answer to this question all coupled one-loop renormalization group equations must be carefully studied for this model.