1 Introduction

It is widely recognized that the Standard Model (SM) is unable to provide explanations to some observed phenomena, such as the extensive evidence supporting the existence of dark matter [1, 2], the neutrinos masses and their mixing [3,4,5,6,7,8], and baryonic asymmetry [9, 10]. In addition to these issues, the recent measurement of the Higgs mass [11, 12] places the SM near the metastability and stability boundaries [13,14,15]. In these studies, the SM is extrapolated to the Planck scale using the two-loop effective potential to analyze the quartic coupling and its beta function in the proximity of this high-energy regime. For the Higgs mass value of \(m_h=125.25\pm 0.17\) GeV [16,17,18], the authors identified an instability scale significantly below Planckian physics, approximately at \(10^{10}\) GeV. According to their interpretation, this finding could be an indicative of the presence of new physics around or below this scale.Footnote 1

In the context of the SM, classical stability is mostly examined through the quartic terms of the Higgs field, h, which for very large values of h takes the form

$$\begin{aligned} V(h)\approx \frac{\lambda }{4}h^4. \end{aligned}$$

Therefore, in this form, the classical potential is stable (bounded from below, BFB) if \(\lambda > 0\). As shown in Ref. [21], \(\lambda \) could not be greater than \(4\pi \), since perturbative theory would break down, nor be very small, since radiative corrections become important. However, the classical potential is found to be insufficient for analyzing the stability of the SM across diverse energy scales since quantum corrections are typically relevant. To address this limitation, the effective potential is employed. As pointed out in Ref. [22], one can always define an effective potential for the SM up to two loops such that, for \(h\gg v\), it assumes the form

$$\begin{aligned} V_{\text {eff.}}(h)=\frac{\lambda _{\text {eff}}(h)}{4}h^4. \end{aligned}$$

This approximation is valid as the instability scale occurs at energies much larger than the electroweak scale. Given that the effective potential shares the same form as the classical potential, one concludes the effective potential stability is realized for \(\lambda _{\text {eff}}>0\). In conclusion, the examination of the stability of the SM is comparatively straightforward, given that both the classical and effective potentials, up to the second loop order, must satisfy an explicit condition (namely \(\lambda >0\) or \(\lambda _\text {eff}>0\)) that are easily fulfilled.

On the other hand, models beyond Standard Model (BSM) exhibit a more intricate potential [23,24,25,26,27,28,29,30]. Typically, these models contain additional scalar fields that interact among themselves increasing the amount of quartic terms, rendering it difficult to determine classical stability conditions, as well as the analytical mass eigenvalues. The situation is aggravated when attempting to study the corresponding effective potential. The logarithm terms coming from the one-loop corrections depend on ratios of mass eigenvalues, obtained from the Hessian matrix of the tree level potential, and the renormalization scale. In a situation where the mass values are very different, it is not possible to define a single renormalization scale that maintains all logarithmic terms under perturbative control. Such effective potentials are referred to in the literature as multiscale [31,32,33,34,35].

The determination of classical stability conditions for multiscale potentials has been extensively explored, as documented in Refs. [31,32,33,34,35]. Generally, for analytical determination, a simplifying condition is commonly imposed to ensure that the quartic terms of the scalar potential take the form \(\lambda _{ij}\phi ^2_{i}\phi ^2_{j}\), making them essentially biquadratic. Therefore, requiring that the scalar potential be bounded from below as the fields approach infinity amounts to ensuring that the matrix \(\lambda _{ij}\) is copositive (conditionally positive). Furthermore, the use of the orbital space method [36] is a common practice to reduce the number of variables, thus increasing the tractability of the problem.

For the multiscale effective potential, some methods were employed in Refs. [37,38,39,40,41]. These methods introduce different scales of renormalization and, therefore, one must work with partial renormalization group equations. Due to this technical complication, the authors in Ref. [42] developed a method to improve the effective potential for any order, using the renormalization group equations with a single renormalization scale. Notably, it has been applied to analyze the extremum structure of scale-invariant effective potentials in a previous work [43]. An interesting feature of this method is that it allows assessment of improved effective potential using tree-level potential, provided that the parameters remain perturbative. The method involves solving the renormalization group equations for the effective potential, choosing the boundary condition on the hypersurface so that the quantum corrections disappear. Thus, tree-level stability criteria are applicable to the improved effective potential. In other words, it allows us to replace the quartic couplings under classical tree-level stability conditions by the running couplings obtained from solving the equations of the renormalization group. Therefore, it is possible to study the quantum stability conditions for any model, using the copositivity criteria and the single scale renormalization to multiscale effective potential.

We are interested in applying these methods to the economical 331 model [44, 45], belonging to a class of models known as 331 models. The name arises as an abbreviation of the semi-simple gauge group \(\text {SU(3)}_C\times \text {SU(3)}_L\times \text {U(1)}_N\). The electric charge operator is defined as \(Q=T_3-bT_8+N\textbf{1}_{3\times 3}\), where \(T_{3,8}\) are the \(\text {SU}(3)_L\) diagonal generators and N is the charge associated with the abelian group U(1). The allowed values for b are \(\sqrt{3}\) or \(1/\sqrt{3}\). The economical 331 model belongs to the latter value of b. This model, when compared to others 331 models, as in Refs. [23, 24], can accommodate a simple scalar structure consisting of three scalar triplets \(\chi ,\rho ,\eta \) without exotic electric charge. In addition, it is also able to accommodate a right-handed neutrino in the leptonic triplet since the \(b=1/\sqrt{3}\) value allows two electrically neutral components. This property allows massive neutrinos at tree level, although quantum corrections are necessary to make them agree with the experiments data [46]. Another interesting feature is the implementation of the Peccei-Quinn mechanism in order to solve strong CP-problem [47] and the existence of axion dark matter [48, 49].

Our objective is to extend the investigation into vacuum stability initiated in Ref. [50]. In that work, the authors introduced a \(\mathbb {Z}_2\) discrete symmetry to the model, facilitating the expression of the classical quartic scalar potential in a biquadratic form. This formulation enables the application of copositivity criteria [34,35,36], leading to the derivation of seventeen classical stability conditions. Solving these conditions, in conjunction with constraints derived from the first and second derivative tests for the potential minima, has, for the first time, allowed the establishment of analytical constraints for the coupling constants.

In order to improve these constrains, a quantum analysis of the vacuum stability within the 331 economical model becomes imperative. In our approach, we employ the previously mentioned single-scale renormalization method [42] to address the effective potential in the context of the renormalization group equations. Given the elevated complexity of the model, we employ the Mathematica RGBeta package, as outlined in Ref. [51], to compute all required beta functions. Subsequently, we numerically solve the eighteen coupled nonlinear differential equations. This process is essential for identifying the new allowed region for the quartic couplings, ensuring that the effective potential remains bounded from below to the Planck scale. As elaborated in Sect. 4, our findings reveal a significantly diminished region for the allowed values of the coupling constants in comparison to the outcomes presented in Ref. [50]. Moreover, we will explore the interplay among the masses of the exotic quarks and the heaviest scalar.

The rest of the paper is organized as follows. In Sect. 2 we present the generalities of the model with an imposed \(\mathbb {Z}_2\) symmetry, bringing simplicity and allowing the scalar potential to be biquadratic in the fields norms. In Sect. 3, we present the mass spectrum of the model in order to determine or constrain the couplings, such as quartic, gauge and Yukawa couplings. Specifically, the expressions for the CP-even masses will be important to study the stability of the model. Moreover, by incorporating complementary information from the \(Z-Z'\) mixing angle, the lower bound on a new charged gauge boson, and the hierarchy of vacuum expectation values in the model, we have determined the minimum value of the first symmetry breaking parameter, \(v_{\chi }\). In Sect. 4, by applying the tree level stability criteria in the improved effective potential, we obtain interesting regions on the quark-Higgs plane and, in the quartic couplings parameter space. Finally, we present our conclusions in Sect. 5.

2 The generalities of the model

An appealing aspect of 331 models is the partial explanation provided for the number of families, a feature absent in the SM. This distinction arises from the fact that, in the SM, the cancellation of gauge anomalies takes place generation by generation. In contrast, in 331 models, the condition for anomaly cancellation implies that the number of families is either three or a multiple of three.

In the present context of the economical 331 model, requiring absence of gauge anomaly dictates that the fermionic field content must be

$$\begin{aligned} \text {Leptons: } L_{aL}&=\left( \nu _{a},\,e_{a},\,N_{a}^{c}\right) _{L}^{\text {T}}\sim \left( 1,\textbf{3},\,-1/3\right) , \end{aligned}$$
(1)
$$\begin{aligned} \text {Quarks: }Q_{L}&=\left( u_{1},\,d_{1},\,u{}_{4}\right) _{L}^{\text {T}}\sim \left( \textbf{3},\,\textbf{3},\,1/3\right) ,\nonumber \\ Q_{bL}&=\left( d_{b},\,u_{b},\,d_{b+2}\right) _{L}^{\text {T}}\sim \left( \textbf{3},\, \bar{\textbf{3}},\,0\right) , \end{aligned}$$
(2)

for the left-handed fields, whereas for the right-handed fields

$$\begin{aligned} \text {Leptons: }e_{aR}&\sim \left( 1,\,1,\,-1\right) , \end{aligned}$$
(3)
$$\begin{aligned} \text {Quarks:\,\ }u_{sR}&\sim \left( 3,\,1,\,2/3\right) ,\quad d_{tR}\sim \left( 3,\,1,\,-1/3\right) , \end{aligned}$$
(4)

where \(a=1,2,3\), \(b=2,\,3\), \(s=1,\dots ,4\), \(t=1,\dots ,5\) and “\(\sim \)” means the transformation properties under the local gauge group.

In order to completely break the gauge symmetry down to \(\text {U}\left( 1\right) _{Q}\) and, at the same time, to give the phenomenologically appropriate masses for all particles, it is necessary to include three scalar triplets in the fundamental representation [52],

$$\begin{aligned} \rho&=\left( \rho _{1}^{+},\,\rho _{2}^{0},\,\rho _{3}^{+}\right) ^{\text {T}}\sim \left( 1,\,\textbf{3},\,2/3\right) ,\nonumber \\ \eta&=\left( \eta _{1}^{0},\,\eta _{2}^{-},\,\eta _{3}^{0}\right) ^{\text {T}}\sim \left( 1,\,\textbf{3},\,-1/3\right) , \end{aligned}$$
(5)
$$\begin{aligned} \chi&=\left( \chi _{1}^{0},\,\chi _{2}^{-},\,\chi _{3}^{0}\right) ^{\text {T}}\sim \left( 1,\,\textbf{3},\,-1/3\right) . \end{aligned}$$
(6)

The introduction of a \(\mathbb {Z}_2\) discrete symmetry, acting on fields as \(\chi \rightarrow -\chi \), \(u_{4R}\rightarrow -u_{4R}\), and \(d_{\left( 4,5\right) R }\rightarrow -d_{\left( 4,5\right) R}\), while leaving the other fields unaffected, not only simplifies the model but also mitigates FCNC problems [53]. Additionally, it facilitates the implementation of the PQ mechanism [47] and, in certain scenarios, contributes to the stabilization of potential dark matter candidates [48, 54, 55].

Taking into consideration all the fermionic and bosonic fields and their symmetries, the most general renormalizable Yukawa Lagrangian reads

$$\begin{aligned} \mathcal {L}_{\text {Yuk}}=\mathcal {L}^{\rho }+\mathcal {L}^{\eta }+\mathcal {L}^{\chi }, \end{aligned}$$
(7)

with

$$\begin{aligned} -\mathcal {L}^{\rho }&= \alpha _{a}\bar{Q}_{L}d_{aR}\rho +\alpha _{ba} \bar{Q}_{bL}u_{aR}\rho ^{*}\nonumber \\ {}&\quad +\text {Y}_{aa'} \varepsilon _{ijk}\left( \bar{L}_{aL}\right) _{i}\left( L_{a'L} \right) _{j}^{c}\left( \rho ^{*}\right) _{k}\nonumber \\ {}&\quad +\text {Y}'_{aa'}\bar{L}_{aL}e_{a'R}\rho + \text {H.c.,} \end{aligned}$$
(8)
$$\begin{aligned} -\mathcal {L}^{\eta }&= \beta _{a}\bar{Q}_{L}u_{aR}\eta +\beta {}_{ba}\bar{Q}_{bL}d_{aR}\eta ^{*}+\text {H.c.}, \end{aligned}$$
(9)
$$\begin{aligned} -\mathcal {L}^{\chi }&= \gamma _{4}\bar{Q}_{L}u_{4R}\chi +\gamma {}_{b\left( b+2\right) } \bar{Q}_{bL}d_{\left( b+2\right) R}\chi ^{*}+\text {H.c.}, \end{aligned}$$
(10)

where \(\epsilon _{ijk}\) is the Levi-Civita symbol and \(a^\prime ,i,j,k=1,2,3\) and a, b, s, t are in the same range as in Eqs. (1–4). Additionally, it is noteworthy that the \(u_{4R}\) and \(d_{(4,5)R}\) quarks exclusively couple to the \(\chi \) triplet. This distinction allows us to identify the \(\chi \) field as the responsible for the initial spontaneous symmetry breaking, whereas \(\eta \) and \(\rho \) will play a role in the subsequent breaking, as we will soon elucidate.

In addition, the most general and renormalizable scalar potential invariant under the gauge and \(\mathbb {Z}_2\) symmetries is

$$\begin{aligned} V\left( \eta ,\rho ,\chi \right)= & {} -\mu _{1}^{2}\eta ^{\dagger }\eta -\mu _{2}^{2}\rho ^{\dagger }\rho -\mu _{3}^{2}\chi ^{\dagger }\chi \nonumber \\{} & {} +\lambda _{1}\left( \eta ^{\dagger }\eta \right) ^{2}+\lambda _{2}\left( \rho ^{\dagger }\rho \right) ^{2}+\lambda _{3}\left( \chi ^{\dagger }\chi \right) ^{2}\nonumber \\ {}{} & {} +\lambda _{4}\left( \chi ^{\dagger }\chi \right) \left( \eta ^{\dagger }\eta \right) \nonumber \\{} & {} +\lambda _{5}\left( \chi ^{\dagger }\chi \right) \left( \rho ^{\dagger }\rho \right) +\lambda _{6}\left( \eta ^{\dagger }\eta \right) \left( \rho ^{\dagger }\rho \right) \nonumber \\ {}{} & {} +\lambda _{7}\left( \chi ^{\dagger }\eta \right) \left( \eta ^{\dagger }\chi \right) +\lambda _{8}\left( \chi ^{\dagger }\rho \right) \left( \rho ^{\dagger }\chi \right) \nonumber \\{} & {} +\lambda _{9}\left( \eta ^{\dagger }\rho \right) \left( \rho ^{\dagger }\eta \right) +\lambda _{10}\left( \chi ^{\dagger }\eta \right) ^{2}\nonumber \\ {}{} & {} -\frac{\lambda _{15}}{\sqrt{2}}\epsilon _{ijk}\eta _{i}\rho _{j}\chi _{k}+\text {H.c.}. \end{aligned}$$
(11)

Note that the term \(\lambda _{15}\epsilon _{ijk}\eta _{i}\rho _{j}\chi _{k}\) effectively breaks the \(\mathbb {Z}_2\) symmetry, but must be included. Its absence would lead to the emergence of a QCD axion with a small decay constant, \(11.5\text { keV}\le f_{a}\le 246\text { GeV}\) [56], already ruled out by experiments [57]. Furthermore, we use the freedom of definition of scalar fields to make \(\lambda _{10}\) and \(\lambda _{15}\) positive real numbers.

3 Mass spectrum

To study the vacuum stability at the one-loop level, we must first determine the mass spectrum. For this reason, in this section we find the analytical expression, at tree level, for particle masses in the model. The minimal conditions for giving mass to all particles is that the scalar fields gain the following vacuum expectation values (VEVs)

$$\begin{aligned}{} & {} \langle \rho \rangle =\frac{1}{\sqrt{2}}(0,\,v_{\rho },\,0)^{\text {T}},\ \langle \eta \rangle =\frac{1}{\sqrt{2}}(v_{\eta },\,0,\,0)^{\text {T}},\nonumber \\{} & {} \langle \chi \rangle =\frac{1}{\sqrt{2}}(0,\,0,\,v_{\chi })^{\text {T}}. \end{aligned}$$
(12)

The first step of the spontaneous symmetry breaking is done by \(v_\chi \), whilst the second is realized by \(v_\rho \) and \(v_\eta \), that is,

$$\begin{aligned} \text {SU(3)}_L\otimes \text {U(1)}_N \xrightarrow {v_\chi }\text {SU(2)}_L\otimes \text {U(1)}_Y\xrightarrow {v_\eta , \, v_\rho }\text {U(1)}_Q. \end{aligned}$$

The starting point of our analysis will be the gauge sector, because, besides being the simplest, it will allow us to estimate VEVs and gauge couplings. Following standard procedures, we define the covariant derivatives as \(D_{\mu }=\partial _{\mu }-ig_L\,W^{a}_{\mu }\,\lambda ^{a}/2-ig_NN\,B_{\mu }\), where \(\lambda ^{a}\) are the Gell-Mann matrices and \(g_L\) and \(g_N\) are the gauge couplings of the SU\((3)_L\) and U\((1)_N\) groups, respectively. After some straighforward calculations, we find that the non-hermitian charged gauge bosons and their respective masses areFootnote 2

$$\begin{aligned}{} & {} W^{\pm }_{\mu }=\frac{1}{\sqrt{2}}(W^1_{\mu }\mp iW^2_{\mu }),\nonumber \\{} & {} X^{0(*)}_{\mu }=\frac{1}{\sqrt{2}}(W^4_{\mu }\mp iW^5_{\mu }),\nonumber \\{} & {} Y^{\pm }_{\mu }=\frac{1}{\sqrt{2}}(W^6_{\mu }\pm iW^7_{\mu }), \end{aligned}$$
(13)
$$\begin{aligned}{} & {} M^2_{W^{\pm }}=\frac{g_L^2}{4}(v_{\eta }^2+v_{\rho }^2),\nonumber \\{} & {} M^2_{X^{0(*)}}=\frac{g_L^2}{4}(v_{\eta }^2+v_{\chi }^2),\nonumber \\{} & {} M^2_{Y^{\pm }}=\frac{g_L^2}{4}(v_{\rho }^2+v_{\chi }^2). \end{aligned}$$
(14)

Note that the new gauge bosons \(X^{0(*)}_{\mu }\) and \(Y^{\pm }_{\mu }\) gain masses at the first step of spontaneous symmetry breaking, and a mass correction coming from the second step. From the experimental value for the SM W gauge boson, \(M_{W^{\pm }}=80.377\pm 0.012\) GeV [58, 59], we calculate that \(v^2_{\text {SM}}\equiv v_{\eta }^2+v_{\rho }^2=(246~\text {GeV})^2\) and \(g_L\simeq 0.66\).

Note that to determine the masses of the \(X^{0(*)}_{\mu }\) and \(Y^{\pm }_{\mu }\) gauge bosons, knowledge of the value of \(v_{\chi }\) is essential. To estimate this value, we consider the mixing angle between Z and \(Z'\) gauge bosons. To achieve this, we first calculate the masses and eigenstates of the hermitian neutral bosons. In addition to the photon mass, \(M_\gamma =0\), and its eigenstate \(A_\mu =\frac{1}{N_\gamma }(2\sqrt{3}\,B_\mu +\sqrt{ 3 }g_L\,g_N\,W_\mu ^3-g_N\,W_\mu ^8)\) (where \(N_\gamma \) is the normalization constant), it is possible to write analytical masses for the other two neutral bosons, \(Z_1\) and \(Z_2\), as

$$\begin{aligned}{} & {} M_{Z_1}^2,\,M_{Z_2}^2\nonumber \\{} & {} \quad =\frac{1}{2}\left[ M_Z^2+M_{Z'}^2\mp \sqrt{(M_Z^2-M_{Z'}^2)^2+(2M_{ZZ'}^2)^2}\right] ,\nonumber \\ \end{aligned}$$
(15)

where

$$\begin{aligned} M_Z^2= & {} \frac{g_L^2}{4\cos ^2\theta _W}\,v_{\text {SM}}^2,\nonumber \\ M_{Z'}^2= & {} \frac{g_L^2}{12(1-\frac{1}{3}\tan ^2\theta _W)}\bigg [(1+\tan ^2\theta _W)^2 v_{\text {SM}}^2\nonumber \\{} & {} -4\tan ^2\theta _W v_{\eta }^2+4v_{\chi }^2\bigg ],\nonumber \\ M_{ZZ'}^2= & {} -\frac{g_L^2}{4\cos ^2\theta _W}\frac{v_{\text {SM}}^2-2\cos ^2 \theta _W v_{\eta }^2}{\sqrt{3-4\sin ^2\theta _W}}. \end{aligned}$$
(16)

The angle \(\theta _W\) in Eq. (16) is the Weinberg angle, measured as \(\sin ^2 \theta _W=0.23122\pm 0.00004\) in the MS-bar scheme [60]. The change of basis from \(Z_1-Z_2\) to \(Z-Z'\) is realized by an SO(2) transformation, with an angle defined as \(\theta _{Z}=\frac{1}{2}\tan ^{-1}\left( \frac{2M^2_{ZZ'}}{M^2_Z-M^2_{Z'}}\right) \). To estimate the values of \(v_{\eta }\) and \(v_{\chi }\), we begin by noting that \(M_Z\), the mass of the SM gauge neutral boson, is measured to be \(91.1876 \pm 0.0021\) GeV [61]. The mixing angle \(\theta _Z\) is constrained to \(-3.98\times 10^{-3} \lesssim \theta _{Z} \lesssim 1.31\times 10^{-4}\) [62]. With this information, we identify the allowed region for \(v_{\eta }\) and \(v_{\chi }\), as depicted in Fig. 1.

Fig. 1
figure 1

The allowed values for \(v_{\eta }\) and \(v_{\chi }\) are determined by considering \(M^2_{ZZ'},\,M^2_{Z}\), and \(M^2_{Z'}\) as functions of these VEVs. It is important to note that, for values of \(v_{\chi }\gtrsim \) 16.71 TeV, there are no restrictions on the values of \(v_{\eta }\)

Full size image

Further considerations involve the lower bound on the mass of a new charged gauge boson, set at 6 TeV at a 95% C.L. according to experimental searches [63, 64]. In this current context, the mass of the \(Y^{\pm }\) boson defined in Eq. (14) can be approximated as \(M^2_{Y^{\pm }} \approx g_L^2\,v_{\chi }^2/4\), assuming \(v_{\chi }\gg v_{\eta }\), which is natural in this model. Finally, employing the previously calculated value of \(g_L=0.66\) and the approximate \(M^2_{Y^{\pm }}\) expression, we can estimate \(v_{\chi }\gtrsim \) 18.1 TeV. Finally, it is noteworthy that for values of \(v_{\chi }\) above or equal to 18.1 TeV, there are no restrictions on the value of \(v_{\eta }\). For the sake of concreteness, we consider the value of \(v_{\eta }\) that sets \(\theta _Z=0\), in other words, the value of \(v_{\eta }\) that sets \(M^2_{ZZ'}\) to zero in Eq. (16). Consequently, we set \(v_{\eta }\approx 197.5\) GeV, and as a result, \(v_{\rho } \approx 147 \) GeV.

In the scalar sector, it is essential to note that the three scalar triplets possess eighteen degrees of freedom. Eight of these correspond to would-be Nambu-Goldstone bosons, which were gauged away to give mass to eight gauge bosons. Thus, we anticipate the presence of ten physical scalar bosons in the mass spectrum. Specifically, the physical scalar spectrum consists of four charged scalars, \(H^{\pm }_1,\ H^{\pm }_2\), four CP-even scalars called h (the 125 GeV SM-like scalar), \(H_0\), H and \(H'\), and two CP-odd scalars \(H_3\) and \(H_5\). Some of the analytical expressions for the masses are relatively straightforward to calculate, as outlined below:

$$\begin{aligned}{} & {} M^2_{H_0}=\frac{v_{\eta }^2+v_{\chi }^2}{2v_{\eta }v_{\chi }}(v_{\rho }\lambda _{15}+v_{\eta }v_{\chi }(\lambda _7+2\lambda _{10})), \end{aligned}$$
(17)
$$\begin{aligned}{} & {} M_{H_{1}^{\pm }}^2=\frac{v_{\text {SM}}^2}{2v_{\eta }v_{\rho }}(\lambda _{15}v_{\chi }+\lambda _9v_{\eta }v_{\rho }),\nonumber \\{} & {} M_{H_{2}^{\pm }}^2=\frac{v_{\rho }^2+v_{\chi }^2}{2v_{\rho }v_{\chi }}(\lambda _{15}v_{\eta }+\lambda _8v_{\rho }v_{\chi }), \end{aligned}$$
(18)
$$\begin{aligned}{} & {} M_{H_3}^2=\frac{v_{\eta }^2+v_{\chi }^2}{2v_{\eta }v_{\chi }}(v_{\rho }\lambda _{15}+v_{\eta }v_{\chi }(\lambda _7- 2\lambda _{10})), \end{aligned}$$
(19)
$$\begin{aligned}{} & {} M_{H_5}^2=\frac{\lambda _{15}}{2v_{\eta }v_{\rho }v_{\chi }}(v_{\eta }^2v_{\rho }^2+v_{\text {SM}}^2 v_{\chi }^2). \end{aligned}$$
(20)

In this study, we will assume \(\lambda _{10}=0\). It is important to highlight that this condition not only simplifies the model significantly but also induces a mass degeneracy between \(H_0\) and \(H_3\). Under this assumption, \(H_0\) and \(H_3\) serve as the real and imaginary components of a complex neutral scalar, respectively.

Concerning the other CP-even scalars (h, H, and \(H'\)), obtaining exact analytical expressions for their masses proves to be overly intricate and not particularly practical. However, we can take advantage of the hierarchy \(v_\chi \gg v_\rho ,\,v_\eta \) to determine approximate analytical expressions by using a perturbative approach [65]. To achieve this, we express \(\lambda _{15}\) as \(\kappa \, v_{\chi }\), where \(\kappa \ll 1\), considering that \(\lambda _{15}\) is a small and dimensionful coupling constant.

More specifically, these CP-even scalars reside in the subspace with basis \((\text {Re}\,\eta ^{0}_1,\) \( \text {Re}\,\rho ^{0}_2, \text {Re}\,\chi ^{0}_3)\), and their squared masses are determined by the matrix

$$\begin{aligned}&M^2_{\text {CP-even}}\nonumber \\&\quad \!\!= \begin{pmatrix} 2\lambda _{1}v^2_{\eta }+\kappa \frac{v_{\rho }v_{\chi }^2}{2v_{\eta }} &{} \lambda _{6}v_{\eta }v_{\rho }-\kappa \frac{v_{\chi }^2}{2} &{} \lambda _{4}v_{\eta }v_{\chi }-\kappa \frac{v_{\rho }v_{\chi }}{2}\\ \lambda _{6}v_{\eta }v_{\rho }-\kappa \frac{v_{\chi }^2}{2} &{} 2\lambda _{2}v_{\rho }^2+\kappa \frac{v_{\eta }v_{\chi }^2}{2v_{\rho }} &{} \lambda _{5}v_{\rho }v_{\chi }-\kappa \frac{v_{\eta }v_{\chi }}{2}\\ \lambda _{4}v_{\eta }v_{\chi }-\kappa \frac{v_{\rho }v_{\chi }}{2} &{} \lambda _{5}v_{\rho }v_{\chi }-\kappa \frac{v_{\eta }v_{\chi }}{2} &{} 2\lambda _{3}v_{\chi }^2+\kappa \frac{v_{\eta }v_{\rho }}{2} \end{pmatrix}. \end{aligned}$$
(21)

The key point is to notice that the exact matrix in Eq. (21), when applied to the VEV hierarchy, can be decomposed into

$$\begin{aligned} M_{\text {CP-Even}}^2/v_\chi ^2 =M_{0}^{2}+M_{1}^{2}+M^2_{2}, \end{aligned}$$
(22)

where

$$\begin{aligned} M_{0}^2&= \begin{pmatrix} \frac{\kappa \,v_{\rho }}{2v_{\eta }} &{} -\frac{\kappa }{2} &{} 0 \\ -\frac{\kappa }{2} &{} \frac{\kappa \,v_{\eta }}{2v_{\rho }} &{} 0 \\ 0 &{} 0 &{} 2\lambda _3 \end{pmatrix},\nonumber \\ M_{1}^{2}&= \begin{pmatrix} 0 &{} 0 &{} \lambda _4\varepsilon _2-\frac{\kappa }{2}\varepsilon _1 \\ 0 &{} 0 &{} \lambda _5\varepsilon _1-\frac{\kappa }{2}\varepsilon _2 \\ \lambda _4\varepsilon _2-\frac{\kappa }{2}\varepsilon _1 &{} \lambda _5\varepsilon _1-\frac{\kappa }{2}\varepsilon _2 &{} 0 \end{pmatrix}\ , \end{aligned}$$
(23)
$$\begin{aligned} M_{2}^{2}&= \begin{pmatrix} 2\lambda _1\varepsilon _2^2 &{} \lambda _6\varepsilon _1\varepsilon _2 &{} 0 \\ \lambda _6\varepsilon _1\varepsilon _2 &{} 2\lambda _2\varepsilon _1^2 &{} 0 \\ 0 &{} 0 &{} \frac{\kappa }{2}\varepsilon _1\varepsilon _2 \end{pmatrix},\ \end{aligned}$$
(24)

with \(\varepsilon _1=\frac{v_{\rho }}{v_{\chi }}\ll 1\) and \(\varepsilon _2=\frac{v_{\eta }}{v_{\chi }}\ll 1\). By straightforwardly applying standard perturbation theory up to the order of \(\varepsilon _{1}^2\) and \(\varepsilon _{2}^2\), we derive the following analytical expressions for the masses:

$$\begin{aligned} m^2_{h}\approx&\frac{2}{v^2_{\text {SM}}}(\lambda _{1}v^4_{\eta }+\lambda _2v_{\rho }^4+\lambda _6v_{\eta }^2v_{\rho }^2)\nonumber \\ {}&\quad -\frac{1}{2\lambda _3 v^2_{\text {SM}}}\left( \lambda _4v_{\eta }^2+\lambda _5v_{\rho }^2-\frac{\lambda _{15}v_{\eta }v_{\rho }}{v_{\chi }}\right) ^2, \end{aligned}$$
(25)
$$\begin{aligned} m^2_{H}&\approx \frac{\lambda _{15}v^2_{\text {SM}}v_{\chi }}{2v_{\eta }v_{\rho }} +\frac{2v_\eta ^2v_\rho ^2}{v^2_{\text {SM}}}\left( \lambda _1+\lambda _2-\lambda _6\right) \nonumber \\ {}&\quad +\frac{v_\eta v_\rho \left[ 2\left( \lambda _4-\lambda _5\right) v_\eta v_\rho v_\chi +\lambda _{15} \left( v_\eta ^2-v_\rho ^2\right) \right] ^2}{2v^2_{\text {SM}}v_\chi \left( \lambda _{15}v^2_{\text {SM}} -4\lambda _3v_\eta v_\rho v_\chi \right) }, \end{aligned}$$
(26)
$$\begin{aligned} m^2_{H'}&\approx 2\lambda _3v^2_{\chi }+\frac{\lambda _{15}v_{\eta }v_{\rho }}{2v_{\chi }} -\frac{\lambda _4^2\lambda _{15}v_\eta ^4v_\chi ^2}{A}-\frac{\lambda _5^2\lambda _{15}v_\rho ^4v_\chi ^2}{A} \nonumber \\ {}&\quad +\frac{v_\eta ^3v_\rho v_\chi \left[ 4\lambda _3\lambda _4^2v_\chi ^2+\left( \lambda _3+2\lambda _4\right) \lambda _{15}^2\right] }{A}\nonumber \\ {}&\quad +\frac{v_\eta v_\rho ^3 v_\chi \left[ 4\lambda _3\lambda _5^2v_\chi ^2+\left( \lambda _3 +2\lambda _5\right) \lambda _{15}^2\right] }{A}\nonumber \\ {}&\quad -\frac{\lambda _{15}v_\eta ^2v_\rho ^2\left[ \left( \lambda _4\lambda _5+2\lambda _3\lambda _4+ 2\lambda _3\lambda _5\right) 2v_\chi ^2+\lambda _{15}^2\right] }{A}, \end{aligned}$$
(27)

where A is defined as

$$\begin{aligned} A=2\lambda _3v^2_{\chi }(4\lambda _3v_{\eta }v_{\rho }v_{\chi }-\lambda _{15}v^2_{\text {SM}}). \end{aligned}$$

These approximate expressions in Eqs. (25)–(27) exhibit only a 3% relative error when compared to the exact masses in a numerical analysis, which is sufficient for our purposes. Additionally, note that \(m_{H'}> m_{H} > m_{h}\) due to the hierarchy between the vacuum expectation values and \(\lambda _3 > \lambda _{15} / v_\chi \).

Another important point is that, at this level of approximation, the SM-like Higgs h and H are given by

$$\begin{aligned}{} & {} h\,=\,\cos {\theta }\,(\text {Re}\,\rho ^0_2)+\sin {\theta }\,(\text {Re}\,\eta ^0_1), \end{aligned}$$
(28)
$$\begin{aligned}{} & {} H\,=-\sin {\theta }\, (\text {Re}\,\rho ^0_2)+\cos {\theta }\,(\text {Re}\,\eta ^0_1), \end{aligned}$$
(29)

where \(\sin {\theta }=v_{\eta }/v_{\text {SM}}\) and \(\cos {\theta }=v_{\rho }/v_{\text {SM}}\). This implies that the coupling between h and the SM leptons, originating from \(\text {Y}'_{aa'}\overline{L}_{aL}\rho e_{a'R}\) in Eq. (8), can be expressed, after rewriting the symmetry eigenstates \(\text {Re}\,\rho ^0_2\) and \(\text {Re}\,\eta ^0_1\) in terms of the mass eigenstates h and H, as

$$\begin{aligned} \mathcal {L}^{\rho }&\supset \cos \theta \frac{h}{v_\rho } \left( m_e\overline{e}e+m_\mu \overline{\mu }\mu +m_\tau \overline{\tau }\tau \right) \nonumber \\&\supset \frac{v_{\text {SM}}\cos \theta }{v_\rho } \frac{h}{v_{\text {SM}}} \left( m_e\overline{e}e+m_\mu \overline{\mu }\mu +m_\tau \overline{\tau }\tau \right) \nonumber \\&\supset \frac{h}{v_{\text {SM}}} \left( m_e\overline{e}e+m_\mu \overline{\mu }\mu +m_\tau \overline{\tau }\tau \right) . \end{aligned}$$
(30)

Therefore, we can conclude that the h scalar actually behaves, at this level of the approximation, as the SM Higgs.

Finally, additional constraints on the \(\lambda _{i}\) couplings can be derived from the positivity of the squared scalar masses in Eqs. (17)–(20) and from the lower bound obtained in the search for charged scalars which defines \(m_{H^{\pm }}>155\) GeV [66]. This information allows us to establish lower bounds for \(\lambda _{7}\), \(\lambda _{8}\) and \(\lambda _{9}\),

$$\begin{aligned}{} & {} \lambda _{7}\ge -\frac{\lambda _{15}v_{\rho }}{v_{\eta }v_{\chi }},\quad \lambda _{8}\ge \frac{2m^2_{H^{\pm }}}{v^2_{\rho }+v^2_{\chi }}-\frac{\lambda _{15}v_{\eta }}{v_{\rho }v_{\chi }},\nonumber \\{} & {} \lambda _{9}\ge \frac{2m^2_{H^{\pm }}}{v^2_{\text {SM}}} -\frac{\lambda _{15}v_{\chi }}{v_{\eta }v_{\rho }}. \end{aligned}$$
(31)

Regarding the quark sector, the exact mass matrices are

$$\begin{aligned}{} & {} M_{u} = \frac{1}{\sqrt{2}} \begin{pmatrix} \beta _1v_\eta &{} \beta _2v_\eta &{} \beta _3v_\eta &{} 0 \\ \alpha _{21}v_\rho &{} \alpha _{22}v_\rho &{} \alpha _{23}v_\rho &{} 0 \\ \alpha _{31}v_\rho &{} \alpha _{32}v_\rho &{} \alpha _{33}v_\rho &{} 0 \\ 0 &{} 0 &{} 0 &{} \gamma _4v_\chi \end{pmatrix},\\{} & {} M_{d} = \frac{1}{\sqrt{2}} \begin{pmatrix} \alpha _1v_\rho &{} \alpha _2v_\rho &{} \alpha _3v_\rho &{} 0 &{} 0 \\ \beta _{21}v_\eta &{} \beta _{22}v_\eta &{} \beta _{23}v_\eta &{} 0 &{} 0 \\ \beta _{31}v_\eta &{} \beta _{32}v_\eta &{} \beta _{33}v_\eta &{} 0 &{} 0 \\ 0 &{} 0 &{} 0 &{} \gamma _{24}v_\chi &{} 0 \\ 0 &{} 0 &{} 0 &{} 0 &{} \gamma _{35}v_\chi \end{pmatrix}. \end{aligned}$$

Observe that the \(3 \times 3\) sub-matrix pertains to the SM quark sector in both cases, whereas the exotic quarks, within the lower sub-matrix, are diagonal due to \(\mathbb {Z}_2\) symmetry. In a general context, determining \(\alpha \) and \(\beta \) angles involves considering quark masses and their mixings in concordance with experimental data, specifically the measured masses and \(V_{\text {CKM}}\) angles. For the sake of completeness, we provide these values below obtained from Ref. [60]

$$\begin{aligned}&m_{u}=2.16\ \begin{array}{c} +0.49 \\ -0.26 \end{array}\ \text {MeV},\quad m_{d}=4.67\ \begin{array}{c} +0.48 \\ -0.17 \end{array}\ \text {MeV},\nonumber \\&m_{s}=93.4\ \begin{array}{c} +8.6 \\ -3.4 \end{array}\ \text {MeV},\end{aligned}$$
(32)
$$\begin{aligned}&m_{c}=1.27 \pm 0.02\ \text {GeV},\ \ m_{b}=4.18 \begin{array}{c} +0.03 \\ -0.02 \end{array}\ \text {GeV},\nonumber \\&m_{t}=172.69 \pm 0.30\ \text {GeV}\end{aligned}$$
(33)
$$\begin{aligned}&\theta _{12}=13.00^{\circ } \pm 0.04^{\circ },\,\quad \,\, \theta _{23}=2.40^{\circ } \begin{array}{c} +0.05^{\circ } \\ -0.04^{\circ } \end{array},\nonumber \\&\theta _{13}=0.211^{\circ } \pm 0.006^{\circ },\end{aligned}$$
(34)
$$\begin{aligned}&\delta =65.5^{\circ } \pm 1.5^{\circ }. \end{aligned}$$
(35)

As evident from Eqs. (34)–(35), the mixing angles are small, implying an approximate equality between mass and symmetry states. Consequently, in this work, we assume that the matrices are diagonal. Notably, the relatively large mass of the top quark justifies the consideration of its Yukawa coupling as the only one significant within the SM quark sector, as it is common practice. A similar scenario happens in the SM lepton sector, i.e. there are no leptonic Yukawa couplings that can significantly modify the renormalization group equations.

After analyzing the mass spectrum of the model, we can now proceed to the study of the stability of the scalar potential at the one-loop level, which we will undertake in next section.

4 Vacuum stability at one-loop level

To investigate the stability of the model at the one-loop level, it is important to revisit the tree level analysis, namely the classical vacuum stability. In the study presented in Ref. [50], classical conditions to guarantee the lower bound of the scalar potential were derived using the method of orbit space along with the copositivity conditions, as detailed in Refs. [34,35,36]. In summary, this method involves formulating invariants as functions of the fields and their norms, represented by the orbit space parameters \(\varvec{\theta }=\frac{f_{ijkl}\phi ^{*}_{i}\phi _{j}\phi ^{*}_{k}\phi _{l}}{|\phi ^{*}_{a}\phi _{a}|^2}\), where a sum over repeated indices is implied. Therefore, all the requisite information to determine the potential minima is encapsulated within these parameters. Consequently, to determine the scalar potential minima for all possible directions in the field space, especially in the large field limit, it suffices to focus on the quartic terms, which is

$$\begin{aligned} V_4= & {} \lambda _1 |\eta |^4 +\lambda _2 | \rho |^4 +\lambda _3| \chi |^4 + (\lambda _4 + \lambda _7 \varvec{\theta }_1 +|\lambda _{10}| \varvec{\theta }_4) |\nonumber \\{} & {} \times \eta |^2|\chi |^2 + (\lambda _5 +\lambda _8 \varvec{\theta }_2) |\rho |^2 |\chi |^2\nonumber \\{} & {} + (\lambda _6 +\lambda _9 \varvec{\theta }_3) |\eta |^2|\rho |^2, \end{aligned}$$
(36)

where the orbit space parameters are

$$\begin{aligned}{} & {} \varvec{\theta }_1(\hat{\eta },\hat{\chi }) = \hat{\chi }^*_j\hat{\eta }_j\hat{\eta }^*_i\hat{\chi }_i=|\varvec{\theta }_1|,\nonumber \\{} & {} {\varvec{\theta }}_2 (\hat{\rho },\hat{\chi }) = \hat{\chi }^*_j\hat{\rho }_j\hat{\rho }^*_i\hat{\chi }_i=|\varvec{\theta }_2|, \end{aligned}$$
(37)
$$\begin{aligned}{} & {} {\varvec{\theta }}_3 (\hat{\eta },\hat{\rho }) = \hat{\eta }^*_i\hat{\rho }_i \hat{\rho }^*_j\hat{\eta }_j=|\varvec{\theta }_3|,\nonumber \\{} & {} \varvec{\theta }_4(\hat{\eta },\hat{\chi }) = 2|\varvec{\theta }_1|\cos {\omega _{\theta _1}}=-2|\varvec{\theta }_1|. \end{aligned}$$
(38)

Note that for \(\varvec{\theta }_4\) we used \(\omega _{\theta _1}=\pi \), which is the correct value for the vacuum. One can think of the \(\varvec{\theta }\) frontier as a cube with edge length equal to one. However, as pointed out in [50], the \(\varvec{\theta }\) frontier is given by

$$\begin{aligned}&0 \le |\varvec{\theta }_{1}|\le 1, \quad 0 \le |\varvec{\theta }_{2}|\le 1, \nonumber \\&\max \left[ 0,\,\sqrt{|\varvec{\theta }_{1}||\varvec{\theta }_{2}|}-\sqrt{\left( 1-|\varvec{\theta }_{1}|\right) \left( 1-|\varvec{\theta }_{2}|\right) } \right] ^2 \le |\varvec{\theta }_{3}| \nonumber \\ {}&\quad \le \left[ \sqrt{|\varvec{\theta }_{1}||\varvec{\theta }_{2}|}+\sqrt{\left( 1-|\varvec{\theta }_{1}|\right) \left( 1-|\varvec{\theta }_{2}|\right) } \right] ^2. \end{aligned}$$
(39)

Notice that \(V_4\) can be expressed in a biquadratic form involving the norms of the fields, represented as \(\textbf{h}^\text {T}\Lambda \textbf{h}\), where \(\textbf{h}^{\text {T}}=(|\eta |^2,\, |\rho |^2,\, |\chi |^2)\ge 0\) and, the matrix \(\Lambda \) is

$$\begin{aligned} \Lambda = \begin{pmatrix} \lambda _1 &{} (\lambda _6 +\lambda _9 \varvec{\theta }_3)/2 &{} (\lambda _4 + \lambda _7 \varvec{\theta }_1 -2\lambda _{10} \varvec{\theta }_1) /2\\ \star &{}\lambda _2 &{} (\lambda _5 +\lambda _8 \varvec{\theta }_2)/2\\ \star &{} \star &{} \lambda _3 \end{pmatrix}. \end{aligned}$$
(40)

Therefore, the scalar potential is bounded from below, if the symmetric matrix \(\Lambda \) is strictly copositive. The use of strong stability requirement, \(V_4>0\), is essential to accommodate the \(\lambda _{15}\) term, as the requirement in the marginal sense, \(V\ge 0\), forbids cubic terms.

By applying the copositivity conditions to the \(\Lambda \) matrix,Footnote 3 one determines seventeen inequalities that serve to constrain the allowed values of \(\lambda _{i}\) couplings,

$$\begin{aligned}&\lambda _1> 0,\quad \lambda _2>0,\quad \lambda _3 >0, \end{aligned}$$
(41)
$$\begin{aligned}&\lambda _4 +2\sqrt{\lambda _1 \lambda _3}>0, \quad \lambda _4 + \lambda _7 -2 |\lambda _{10}|+2\sqrt{\lambda _1 \lambda _3}>0, \end{aligned}$$
(42)
$$\begin{aligned}&\lambda _5 + 2\sqrt{\lambda _2 \lambda _3}>0,\quad \lambda _5 +\lambda _8 + 2\sqrt{\lambda _2 \lambda _3}>0, \end{aligned}$$
(43)
$$\begin{aligned}&\lambda _6 + 2\sqrt{\lambda _1 \lambda _2}>0, \quad \lambda _6 +\lambda _9 + 2\sqrt{\lambda _1 \lambda _2}>0, \end{aligned}$$
(44)
$$\begin{aligned}&\text {C}_{1}\sqrt{\lambda _2}+\text {C}_{2}\sqrt{\lambda _1}+\text {C}_{3}\sqrt{\lambda _3}+2\sqrt{\lambda _1 \lambda _2 \lambda _3}+\sqrt{\overline{\text {C}}_{1}\overline{\text {C}}_{2}\overline{\text {C}}_{3}}{>} 0, \end{aligned}$$
(45)

where \( \text {C}_1=\lbrace \lambda _4,\,\lambda _4 + \lambda _7 -2 |\lambda _{10}|\rbrace \), \( \text {C}_2=\lbrace \lambda _5,\,\lambda _5 +\lambda _8 \rbrace \), \( \text {C}_3= \lbrace \lambda _6,\,\lambda _6 +\lambda _9 \rbrace \), \(\overline{\text {C}}_1= \text {C}_1+2\sqrt{\lambda _1\lambda _{3}}\), \( \overline{\text {C}}_2=\text {C}_2+2\sqrt{\lambda _2\lambda _{3}}\) and \(\overline{\text {C}}_3=\text {C}_3+2\sqrt{\lambda _1\lambda _{2}}\). It is important to emphasize that the conditions of equations (41)–(45) are necessary but not sufficient to ensure the boundedness from below of the scalar potential. Necessary and sufficient conditions should include an additional condition similar to Eq. (45), concerning a minimized inequality over all \(\theta _i\) simultaneously on the orbit space surface given by Eq. (39). However, Eqs. (41)–(45) serve as an excellent approximation to the complete conditions.

Now, our attention shifts to the examination of model stability at the one-loop level. It is well-established that, at this order, numerous interactions not present in the original Lagrangian may manifest, giving rise to substantial modifications in the stability characteristics of the model. Not only that, but at one-loop level, the parameters acquire an energy scale-dependence, as a consequence of the renormalization procedure, and the parameters are referred to as running couplings. This procedure generates first order differential equations known as Renormalization Group Equations, or RGE, for short. It could tell us how the parameters change, when a change in scale is made.

For models like the SM and extensions, such as 331 models, it is not an easy task to determine and solve the RGE, due to the large number of Feynman diagrams associated to the large number of parameters. Fortunately, there are numerous softwares to compute these one loop diagrams and the RGE, see for instance Ref. [51, 67,68,69]. In our analysis, we have used RGBeta [51]. This package can provide calculations up to four-loop order for the gauge couplings and up to the three-loop order for the remaining parameters. Nevertheless, for the current work, a calculation at the one-loop level is more than sufficient to illustrate the primary behavior of the model at the quantum level.

The renormalization group equations take on a generic form at the one-loop level, expressed as \(\frac{d{X}}{dt}\equiv \beta _{X}/(4\pi )^2\) (where t is a real parameter defined as \(t=\log {\mu /\mu _0}\), and in this work, we use \(\mu _0=m_Z\)), with X representing a generic coupling. Using RGBeta, we derive the following expressions for the \(\beta \)-functions:

$$\begin{aligned} \beta _{g_N}&= \frac{26}{3}g_N^3, \ \beta _{g_L}=-\frac{13}{2}g_L^3,\ \beta _{g_3}=-5g_3^3, \end{aligned}$$
(46)
$$\begin{aligned} \beta _{\alpha _{33}}&=\alpha _{33}\left( -\frac{4}{3}g_{N}^2-4g_{L}^2-8g_3^2+5\alpha _{33}^2+\frac{1}{2}\gamma _{35}^2\right) , \end{aligned}$$
(47)
$$\begin{aligned} \beta _{\gamma _4}&=\gamma _{4}\left( -\frac{5}{3}g_{N}^2-4g_{L}^2-8g_3^2+5\gamma _{4}^2+3\gamma _{24}^2+3\gamma _{35}^2\right) \end{aligned}$$
(48)
$$\begin{aligned} \beta _{\gamma _{24}}&=\gamma _{24}\left( -\frac{1}{3}g_{N}^2-4g_{L}^2-8g_3^2+3\gamma _{4}^2+5\gamma _{24}^2+3\gamma _{35}^2\right) , \end{aligned}$$
(49)
$$\begin{aligned} \beta _{\gamma _{35}}&=\gamma _{35}\left( -\frac{1}{3}g_{N}^2-4g_{L}^2-8g_3^2+\frac{1}{2}\alpha _{33}^2\right. \nonumber \\&\quad \left. +3\gamma _{4}^2+3\gamma _{24}^2+5\gamma _{35}^2\right) , \end{aligned}$$
(50)
$$\begin{aligned} \beta _{\lambda _1}&=\frac{2}{27}g_{N}^4+\frac{4}{9}g_{N}^2g_{L}^2+\frac{13}{6}g_{L}^4-\left( \frac{4}{3}g_{N}^2+16g_{L}^2\right) \lambda _1\nonumber \\&\quad +28\lambda _1^2+4\lambda _{10}^2 +3\lambda _4^2+3\lambda _6^2+2\lambda _4\lambda _7+\lambda _7^2\nonumber \\&\quad +2\lambda _6\lambda _9+\lambda _9^2, \end{aligned}$$
(51)
$$\begin{aligned} \beta _{\lambda _2}&=\frac{32}{27}g_{N}^4+\frac{16}{9}g_{N}^2g_{L}^2+\frac{13}{6}g_{L}^4-\left( \frac{16}{3}g_{N}^2+16g_{L}^2\right) \lambda _2\nonumber \\&\quad +28\lambda _2^2+3\lambda _5^2+3\lambda _6^2+2\lambda _5\lambda _8+\lambda _8^2+2\lambda _6\lambda _9\nonumber \\&\quad +\lambda _9^2-6\alpha _{33}^4+12\alpha _{33}^2\lambda _2, \end{aligned}$$
(52)
$$\begin{aligned} \beta _{\lambda _3}&=\frac{2}{27}g_{N}^4+\frac{4}{9}g_{N}^2g_{L}^2+\frac{13}{6}g_{L}^4-\left( \frac{4}{3}g_{N}^2+16g_{L}^2\right) \lambda _3\nonumber \\&\quad +28\lambda _3^2+3\lambda _4^2+3\lambda _5^2+2\lambda _4\lambda _7+\lambda _7^2+2\lambda _5\lambda _8\nonumber \\&\quad +\lambda _8^2-6\gamma _{4}^4-6\gamma _{24}^4-6\gamma _{35}^4\nonumber \\&\quad +12\left( \gamma _{4}^2+\gamma _{24}^2+\gamma _{35}^2\right) \lambda _3+4\lambda _{10}^2, \end{aligned}$$
(53)
$$\begin{aligned} \beta _{\lambda _4}&=\frac{4}{27}g_{N}^4-\frac{4}{9}g_{N}^2g_{L}^2+\frac{11}{6}g_{L}^4-\left( \frac{4}{3}g_{N}^2+16g_{L}^2\right) \lambda _4\nonumber \\&\quad +8\lambda _{10}^2+16\lambda _1\lambda _4+16\lambda _3\lambda _4+4\lambda _4^2+6\lambda _5\lambda _6\nonumber \\&\quad +4\lambda _1\lambda _7+4\lambda _3\lambda _7+2\lambda _7^2+2\lambda _6\lambda _8+2\lambda _5\lambda _9\nonumber \\&\quad +6\left( \gamma _{4}^2+\gamma _{24}^2+\gamma _{35}^2\right) \lambda _4, \end{aligned}$$
(54)
$$\begin{aligned} \beta _{\lambda _5}&=\frac{16}{27}g_{N}^4+\frac{8}{9}g_{N}^2g_{L}^2+\frac{11}{6}g_{L}^4-\left( \frac{10}{3}g_{N}^2+16g_{L}^2\right) \lambda _5\nonumber \\&\quad +16\lambda _2 \lambda _5+16\lambda _3\lambda _5 +4\lambda _5^2+6\lambda _4\lambda _6+2\lambda _6\lambda _7\nonumber \\&\quad +4\lambda _2\lambda _8+4\lambda _3\lambda _8+2\lambda _8^2+2\lambda _4\lambda _9\nonumber \\&\quad +6\left( \alpha _{33}^2+\gamma _{4}^2+\gamma _{24}^2+\gamma _{35}^2\right) \lambda _5, \end{aligned}$$
(55)
$$\begin{aligned} \beta _{\lambda _6}&=\frac{16}{27}g_{N}^4+\frac{8}{9}g_{N}^2g_{L}^2+\frac{11}{6}g_{L}^4-\left( \frac{10}{3}g_{N}^2+16g_{L}^2\right) \lambda _6\nonumber \\&\quad +6\lambda _4\lambda _5+16\lambda _1\lambda _6 +16\lambda _2\lambda _6+4\lambda _6^2+2\lambda _5\lambda _7\nonumber \\&\quad +2\lambda _4\lambda _8+4\lambda _1\lambda _9+4\lambda _2\lambda _9+2\lambda _9^2+6\alpha _{33}^2\lambda _6, \end{aligned}$$
(56)
$$\begin{aligned} \beta _{\lambda _7}&=\frac{4}{3}g_{N}^2g_{L}^2+\frac{5}{2}g_{L}^4-\left( \frac{4}{3}g_N^2+16g_L^2\right) \lambda _7\nonumber \\&\quad +40\lambda _{10}^2+4\lambda _1 \lambda _7+4\lambda _3\lambda _7 +8\lambda _4\lambda _7+6\lambda _7^2+2\lambda _8\lambda _9\nonumber \\&\quad +6\left( \gamma _{4}^2+\gamma _{24}^2+\gamma _{35}^2\right) \lambda _7, \end{aligned}$$
(57)
$$\begin{aligned} \beta _{\lambda _8}&=-\frac{8}{3}g_{N}^2g_{L}^2+\frac{5}{2}g_{L}^4-\left( \frac{10}{3}g_{N}^2+16g_{L}^2\right) \lambda _8\nonumber \\&\quad +4\lambda _2\lambda _8+4\lambda _3\lambda _8+8\lambda _5\lambda _8+6\lambda _8^2+2\lambda _7\lambda _9\nonumber \\&\quad +6\left( \alpha _{33}^2+\gamma _{4}^2+\gamma _{24}^2+\gamma _{35}^2\right) \lambda _8-12\gamma _{35}^2\alpha _{33}^2, \end{aligned}$$
(58)
$$\begin{aligned} \beta _{\lambda _9}&=-\frac{8}{3}g_{N}^2g_{L}^2+\frac{5}{2}g_L^4-\left( \frac{10}{3}g_{N}^2+16g_{L}^2\right) \lambda _9+2\lambda _7\lambda _8\nonumber \\&\quad +4\lambda _1\lambda _9+4\lambda _2\lambda _9+8\lambda _6\lambda _9+6\lambda _9^2+6\alpha _{33}^2\lambda _9, \end{aligned}$$
(59)
$$\begin{aligned} \beta _{\lambda _{10}}&=\lambda _{10}\left[ -\frac{4}{3}g_{N}^2-16g_{L}^2+4 \lambda _3+8\lambda _4+4\lambda _1+16\lambda _7\right. \nonumber \\&\quad \left. +6\left( \gamma _{4}^2+\gamma _{24}^2 +\gamma _{35}^2\right) \right] . \end{aligned}$$
(60)
$$\begin{aligned} \beta _{\lambda _{15}}&=\lambda _{15}[-2g^2_{N}-12g^2_L +3(\gamma ^2_{24}+\gamma ^2_{35}+\gamma _{4}^2+\alpha _{33}^2)\nonumber \\&\quad +2(\lambda _{4}+\lambda _5+\lambda _6-\lambda _7-\lambda _8-\lambda _9)] \end{aligned}$$
(61)

As discussed in the previous section, when we adopt \(\lambda _{10}=0\), a new symmetry emerges. This can also be seen from Eq. (60), which vanishes for this choice. Notice that a similar situation occurs for \(\lambda _{15}\), which is a U\((1)_{\text {PQ}}\) symmetry studied in [56], of which the well-known \(\mathbb {Z}_2\) symmetry is a discrete subgroup.

Now, we focus on the issue of the loss of perturbativity in the effective potential at the one-loop level. To do this, let us recall that the effective potential contains logarithmic terms of the ratios of the mass eigenvalues to the renormalization scale. In other words, the effective potential at one-loop level is generically written as:

$$\begin{aligned} V_{\text {eff}}(\mu ,\,\lambda ,\,\phi )= & {} V(\lambda ,\,\phi )+\frac{\hbar }{64\pi ^2}\sum _{i}n_{i}m^4_{i}(\lambda ,\,\phi )\nonumber \\{} & {} \times \left( \ln {\frac{m^2_{i}(\lambda ,\,\phi )}{\mu ^2}}+\zeta _{i}\right) , \end{aligned}$$
(62)

where \(\hbar \) indicates the one-loop level. The i index runs over all particle species. The \(n_i\) is the number of degrees of freedom associated to the field, and \(m_i\) is its mass eigenvalue. In addition, \(\zeta _{i}\) is a constant that depends on the renormalization scheme, having different values depending on the nature of the field. Moreover, we are considering \(\lambda \) and \(\phi \) as a collection of quartic couplings and fields, respectively.

In order to maintain perturbativity, the mass scales can not be very different when the energy scale \(\mu \) increases, since the large logarithms would spoil perturbativity. However, as we have seen in the previous section, the masses of the SM sector have very distinct values to those of the new particles arising from the economical 331 model. This may create a significant problem related to the perturbativity in the current case. For this reason, we use the method in Ref. [42] which improves the effective potential using a single renormalization scale. The key point of this method is to choose a renormalization scale \(\mu =\mu ^{*}\) where \(\mu ^{*}\) is on the hypersurface where the one-loop contributions to the effective potential vanishes, that is, that the effective potential in Eq. (62) is equal to the classical potential evaluated in the running coupling \(\lambda (\mu ^{*})\),

$$\begin{aligned} V_{\text {eff.}}(\mu ^{*},\,\lambda ,\,\phi )=V(\lambda (\mu ^{*}),\,\phi (\mu ^{*})). \end{aligned}$$
(63)

This choice is motivated by the fact that, while it is not possible to individually suppress all logarithms with a single renormalization scale, it is possible to suppress them altogether when evaluating on this hypersurface. One important consequence of this method is the ability to use classical stability conditions to the improved effective potential. In practice, by solving the RGEs, we can substitute the \(\lambda \) on the tree-level stability criteria, Eqs. (41)–(45), by their respective running couplings \(\lambda (\mu ^{*})\).

With everything settled, our goal is to identify constraints on the masses of the new particles introduced by the model. Specifically, we choose to show bounds as functions of the exotic quark mass and the heaviest scalar, \(m_{H'}\), since these particles gain masses predominantly at the 331 scale. Additionally, we aim to uncover the allowed values among the quartic couplings in the parameter space that ensure the stability of the scalar potential at high energies. In order to achieve this, one needs to solve the eighteen RGEs, which are non-linear and coupled, considering a set of initial conditions suitable for the model parameters, which are capable of evolving the entire parameter space up to the Planck scale \(\approx 10^{19}\) GeV. Certain parameters of the economical 331 model have already been determined in Sect. 3 through additional constraints. These constraints include, for example, ensuring the positivity of squared masses for the scalars, considering the \(Z-Z'\) mixing, and taking into account searches for new charged gauge bosons and charged scalars. However, other initial parameters, especially the scalar couplings \(\lambda \)s, are less restricted and need to be chosen through trial and error, ensuring that they at least satisfy the constraints in Eqs. (41)–(45). A set of parameters that satisfies all the aforementioned constraints and allows the economical 331 model to have stable regions up to the Planck scale is as follows:

$$\begin{aligned}&g_{N}=0.374,\quad \quad \ \ \, g_{L}=0.664,\quad \ \,\, g_{3}=1.22, \end{aligned}$$
(64)
$$\begin{aligned}&v_{\eta }=197.52\ \text {GeV},\ v_{\rho }=147\ \text {GeV},\ v_{\chi }=18.1\ \text {TeV}, \end{aligned}$$
(65)
$$\begin{aligned}&\lambda _1=0.1,\ \lambda _2=0.15,\ \lambda _4=0.032,\ \lambda _5=-0.055,\ \lambda _7=0, \end{aligned}$$
(66)
$$\begin{aligned}&\lambda _8=0,\ \lambda _9=-0.5,\ \lambda _{10}=0,\ \lambda _{15}=25\ \text {GeV},\ \alpha _{33}=0.5, \end{aligned}$$
(67)

where \(g_3=\sqrt{4\pi \alpha _s}\), with \(\alpha _s=0.1180 \pm 0.009\) [60]. Furthermore, by using the relation \(\tan {\theta _W}=-\sqrt{3}\sin {\theta _X}\), where \(\theta _X\) is the rotation angle associated to the first symmetry breaking, we find \(\theta _X=-18^{\circ }\). With this value, we are able to use the relation \(e=g_{N}\cos {\theta _X}\cos {\theta _W}\), where e is the elementary charge associated with the U\((1)_{Q}\) symmetry, in order to obtain \(g_{N}=0.374\).

The exotic Yukawa couplings (which we assume to be equal in this work because we have no a priori reason to make the exotic quark masses, \(m_{q_i}\), different) and \(\lambda _3\) are variables in this analysis, since we use them to change the exotic quark mass and the heaviest scalar, \(m_{H'}\). As a consequence of this parametrization, we choose the \(\lambda _6\) coupling to become a function of \(\lambda _{3}\) in order to fix the Higgs mass in Eq. (25) to be the measured value. Hence, by applying the tree-level stability criteria, from Eqs. (41)–(45), up to Planck scale \(\approx 10^{19}\) GeV, we obtain the region in the \(m_{q}-m_{H'}\) plane, shown in Fig. 2.

Fig. 2
figure 2

The economical 331 model vacuum stability for the allowed values of \(m_{H'}\) and \(m_{q}\). The non-perturbativity region is defined by values for which \(\lambda _{i}>4\pi \). The border (black line) between the three regions occurs on the Planck scale \(10^{19}\) GeV. In contrast to the non-perturbative scenario, the stability-instability border exhibits a more rapid convergence of scales. Notably, the non-perturbative boundary diminishes with decreasing scale

Full size image

To understand the boundaries shown in Fig. 2, we need to resort to the RGEs, more specifically to the \(\beta _{\lambda _3}\) function in Eq. (53). There are negative contributions coming from the quartic terms of the exotic Yukawa quark couplings, i.e. \(\gamma ^4_4\), \(\gamma ^4_{24}\), and \(\gamma ^4_{35}\) terms, and positive contributions coming from \(\lambda _3\) and \(\lambda ^2_3\) terms. As \(m_{H'}\) increases, \(\lambda _3\) also increases due to its natural positive contribution. However, as the exotic Yukawa quark couplings increase, \(m_{q}\) increases, leading to a decrease in the \(\beta _{\lambda _3}\) function.

This intricate interplay between the couplings persists until the scalar potential becomes unstable, reaching a maximum value of approximately 14 TeV for \(m_{H'}\) and 11 TeV for \(m_{q}\). The relation between these couplings is further illustrated in Fig. 3, where the behaviour of \(m_q\) as function of \(\log (\mu /\text {GeV})\) is presented. It is clear that for different values of \(m_{H'}\), \(m_{q}\) reaches different maximum values. For instance, when \(m_{H'}=12\) TeV, the maximum value of \(m_q\) is approximately 11 TeV, maintaining it constant for energy scales \(\gtrsim 10^8\) GeV, a detail that may not be immediately apparent from Fig. 2.

Fig. 3
figure 3

The maximum mass value for the exotic quark masses \(m_q\) as function of \(m_{H'}\) and the energy scale \(\mu \). Note that as \(\mu \) approaches to \(m_Z\), which is the renormalization subtraction point, \(m_{q}\) has arbitrarily large values. This is to be expected, since the vacuum is classically stable regardless of \(m_{q}\). Hence, as the effective couplings begin to evolve, so does \(m_{q}\), eventually approaching a stable value

Full size image
Fig. 4
figure 4

The dashed line represents the classical region frontier. The lighter green region depicts the allowed values to the potential be stable up to \(10^{12}\) GeV, while the darker green one the potential is stable up to \(10^{16}\) GeV. Note that the stability up to Planck scale, \(10^{19}\) GeV, is not possible within this scenario

Full size image

Another way to utilize vacuum stability and ensure perturbativity up to the Planck scale to constrain the model is to directly examine the couplings. This approach is particularly relevant as one of these parameters can reflect the intensity of a portal between the SM sector and the new 331 sector. An example of this is observed in dark matter studies through the Higgs portal within the real singlet scalar model [70,71,72]. With this motivation in mind, we conducted an analysis of the parameter space defined by \(\lambda _{4}-\lambda _{5}\) and \(\lambda _{4}-\lambda _{7}\) to impose additional constraints on the allowed region within this domain. In our studies, we observed that if the masses of exotic quarks are below 9 TeV, the one-loop region depicted is nearly identical to the classical one. Therefore, for this particular analysis, we set \(m_{q}=9\) TeV. Furthermore, in this scenario, the dependence of \(m_h\) on \(\lambda _4\) demands the variation of \(\lambda _6\), similar to the approach adopted previously for Fig. 2. Consequently, stability is not observed at the Planck scale but only for energy values below \(10^{16}\) GeV. Another interesting feature of our analysis is the constraint imposed on \(\lambda _7\). In a prior study in Ref. [50], the authors established the maximum and minimum allowed values on the \(\lambda _4-\lambda _7\) plane in the classical case. Our findings indicate that, to uphold the positivity of mass square at the one-loop level, \(\lambda _7\) must be greater than or equal to zero, otherwise, \(M^2_{H_0}\) and \(M^2_{H_3}\) assume a negative value. It is also interesting to compare the vacuum stability region of the economical 331 model at the one-loop level to the classical region obtained in Ref. [50]. Figure 4 clearly shows a significant reduction in the vacuum stability region in the quantum case.

5 Conclusions

In this work, we study vacuum stability at one-loop of the economical 331 model by resumming the renormalization group equations (RGE) into the tree level potential which are obtained from the Mathematica package RGBeta [51]. We apply the method of single scale renormalization for multiscale effective potential developed in [42], allowing us to integrate the RGE and substitute the solutions into the tree level stability criteria determined in Ref. [50]. The method allows any boundaries conditions, as long as the one loop potential vanishes on it. Therefore, one may choose arbitrary conditions for the couplings and vary them on a grid, requiring stability up to any scale. For our purposes, the Planck scale is more than enough, in order to compare to the SM or its extensions. The latest fit data for new Higgs-like bosons, charged scalars, charged and neutral gauge bosons are used to set constraints on the model parameters, namely, the symmetry breaking scale \(v_{\chi }\), for which we determine a minimum value of \(\approx 18.1\) TeV. We also assume a zero mixing angle between \(Z-Z'\) allowing us to determine the values of \(v_{\eta },\ v_{\rho }\).

In the scalar sector, we assumed a global U(1) symmetry emerging when setting the \(\lambda _{10}\) coupling to zero. Similar observations have also been noted in the two Higgs doublet model [73, 74]. For the CP-even scalars, we derived more precise expressions for the scalar masses than those shown in [50]. These expressions were obtained by applying traditional non-degenerate perturbation theory, and their formulations align with those in [75] for the first few terms.

We select an appropriate set of parameters for the \(\lambda _i\) couplings, perform numerical solutions of the RGE up to the Planck scale, and thereby unravel, for the first time, intricate relations between the mass of the heaviest scalar, \(m_{H'}\), and the masses of the exotic quarks, \(m_q\), ensuring the stability of the economical 331 model up to the Planck scale, shown in Fig. 2. These regions looks similar to the one shown in [13], depicting Higgs-top mass plane in the SM context. Remarkably, a very interesting limit for the heaviest exotic quark mass appears in Fig. 2, and more detailed in Fig. 3. The first figure illustrates the maximum values of \(m_{q},\, m_{H'}\) where the model remains stable and perturbative. The latter figure shows an interesting phenomenon: the maximum value for the exotic quark mass stabilizes way before the Planck scale. A similar situation occurs in the analysis of the quark top mass in Ref. [13], stabilizing near \(10^{12}\) GeV, whereas in our case the upper limit on the heaviest exotic quark stabilizes near \(10^{8}\) GeV. Hence, we have determined an upper bound for the heaviest quark mass within the model, which is not so distant from future LHC runs, serving as bounds to be searched.

Finally, we explore interesting relations between the \(\lambda _i\) couplings emerging from the vacuum stability and the perturbativity conditions. In particular, we choose to illustrate the different regions of stability in the \(\lambda _4-\lambda _5\) and \(\lambda _4-\lambda _7\) planes, see Fig. 4. Although, at a first glance, these \(\lambda _i\) parameters are not directly related to a physical observable, the allowed region from this figure is useful to constrain the large set of parameters of the economical 331 model.