Constraining scalar doublet and triplet leptoquarks with vacuum stability and perturbativity

Abstract

We investigate the constraints on the leptoquark Yukawa couplings and the Higgs-leptoquark quartic couplings for scalar doublet leptoquark $\widetilde{R}_2$, scalar triplet leptoquark $\varvec{S}_3$ and their combination with both three generations and one generation with respect to perturbative unitarity and vacuum stability. The perturbative unitarity of all the dimensionless couplings is studied via one- and two-loop beta functions. New $SU(2)_L$ multiplets in terms of these leptoquarks are introduced to fabricate Landau poles at the two-loop level in the gauge coupling $g_2$ at $10^{19.7}$ GeV and $10^{14.4}$ GeV, respectively, for the $\varvec{S}_3$ and $\widetilde{R}_2+\varvec{S}_3$ models with three generations. However, such Landau poles cease to exist for $\widetilde{R}_2$ and any of these extensions with both one and two generations up to Planck scale. The Higgs-leptoquark quartic couplings acquire severe constraints to protect Planck scale perturbativity, whereas leptoquark Yukawa couplings acquire some upper bound in order to respect Planck scale stability of Higgs vacuum. The Higgs quartic coupling at the two-loop level constrains the leptoquark Yukawa couplings for $\widetilde{R}_2,\varvec{S}_3, \,\widetilde{R}_2+\varvec{S}_3$ with values with three generations. In the effective potential approach, the presence of any of these leptoquarks with any number of generations pushes the metastable vacuum of the Standard Model to the stable region.

1 Introduction

Over the past few decades, the Standard Model (SM) has been extremely successful in establishing itself as a well-accepted model providing beautiful theoretical descriptions of elementary particles. After the discovery of the Higgs boson [1, 2], the last undetected particle of the SM, followed by precise measurement of its properties at the Large Hadron Collider (LHC), the particle spectrum of the SM was complete. However, because of the inability to explain various experimental facts such as matter–antimatter asymmetry, dark matter relic density, neutrino masses, Higgs mass hierarchy and several flavour anomalies, SM is considered an incomplete theory. This motivates one to extend the SM with some beyond-Standard Model (BSM) particles or new gauge groups or additional discrete symmetries. Various new physics (NP) models augmented with heavy fermions and bosons have been very well studied in the literature. Leptoquarks [3] lie under the category of bosonic extension of the SM, but with lepton and baryon number.

Though the notion of leptoquark [4, 5] has existed in the literature for nearly 50 years, it has received much greater attention in recent years due to its prospects for addressing various flavour anomalies [6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31], unexplained with SM. Simply speaking, leptoquarks are hypothetical particles having both lepton number and baryon number. They are electromagnetically charged and colour triplet (fundamental or anti-fundamental) under the $SU(3)_C$ gauge group. Under the $SU(2)_L$ gauge group, they can be singlet, doublet and triplet as well. According to Lorentz representation, they might be scalar as well as vector. These leptoquarks emerge naturally in several higher gauge theories unifying matter [4, 5, 32,33,34,35,36,37,38,39,40,41,42]. In the literature, numerous efforts have been devoted to studying the phenomenology of these leptoquarks at colliders [43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73], in particular at the LHC. Mainly focusing on the angular distributions, distinguishing features of scalar and vector leptoquarks carrying different SM gauge quantum numbers have also been explored at electron–proton [74], electron–photon [75] and proton–proton [76, 77] colliders. On the other hand, many experimental searches for these leptoquarks have been performed at electron–positron [78,79,80,81], electron–proton [82, 83], proton-antiproton [84,85,86] and proton–proton [87,88,89,90,91] colliders, but no sign of them has yet been confirmed. Kaon and lepton physics have implemented strong constraints on the coupling of leptoquarks to the first generation of quarks and leptons [3, 92, 93]. ATLAS and CMS (Compact Muon Solenoid) have performed thorough generation-wise analyses on the allowed mass range of scalar and vector leptoquarks. These studies [87, 90, 91] suggest that if any leptoquark exists, it must have mass above 1.5 TeV with the coupling to quarks and leptons below the electromagnetic coupling constant.¹

Now, the 125.5 GeV mass of the observed Higgs boson indicates that its vacuum cannot be completely stable all the way up to Planck scale or even Grand Unified Theory (GUT) scale [94]. In order for the Higgs potential to be bounded from below, the self-quartic-coupling ($\lambda _h$) of the Higgs boson must be positive. However, it is found that the negative quantum correction from the top quark pushes $\lambda _h$ to negative values above the energy scale of $10^{10}$ GeV, thus hampering the stability of the SM. Technically speaking, it is generally considered that the SM is in a metastable state. In these circumstances, the presence of some BSM scalar extensions, i.e. simplest extension via singlet [95,96,97,98,99,100,101,102,103,104], $SU(2)_L$ doublet [105,106,107,108,109,110,111,112] or triplet representation of $SU(2)_L$ [113, 114], is required to restore the stability of vacuum by neutralizing the destabilizing effect of the top quark. On the other hand, the inclusion of additional fermionic particles worsens the case by further lowering the energy scale within which $\lambda _h$ remains positive. To avoid the stability issue, these models are also often extended with additional scalar particles [115,116,117,118,119]. However, it is important to note that fermions with $SU(2)_L$ gauge charge push for non-perturbativity, thus constraining the number of generations for the Planck scale perturbativity. [120]. This motivates us to investigate the stability of the vacuum in the presence of scalar leptoquarks, which has not been well explored so far.

Furthermore, it is expected that every dimensionless parameter of a fundamental model should be bounded from above in order to ensure the perturbative expansion of the correlation functions. Now, the presence of the leptoquark will hamper the perturbativity of the theory by imposing extra contributions on the renormalization group (RG) evolution of different SM couplings. Therefore, it is of paramount importance to examine the perturbativity of a model when studying the stability of its vacuum.

Along with perturbativity, the effects of the scalar singlet leptoquark $S_1$ in addressing the issue of vacuum stability has already been discussed in Ref. [121]. In this paper, we study the stability and perturbativity of the models with scalar triplet leptoquark $\varvec{S}_3$ and scalar doublet leptoquark ${\widetilde{R}}_2$. Since leptoquarks possess colour charge as well as hypercharge, their presence affects the RG evolution of all the couplings quite differently from that of conventional scalars. Moreover, doublet and triplet leptoquarks originate more positive effects, required for stability, than the singlet type, as they contain two and three different components, respectively. On similar grounds, such models are often more constrained by perturbativity. In addition, we study the BSM scenario having both leptoquarks ${\widetilde{R}}_2$ and $\varvec{S}_3$ simultaneously. This model has attracted increasing interest given its prospects for generating the Majorana mass term for neutrinos at one- and two-loop levels, along with other beautiful features [3, 122,123,124,125,126,127,128].

The paper is organized as follows. In the next section (Sect. 2), a brief illustration of all the leptoquark models considered for this paper is presented. Section 3 deals with perturbativity of these models in terms of different gauge couplings, top and leptoquark Yukawa couplings and Higgs-leptoquark quartic couplings. In the following section (Sect. 4), we scrutinize the stability of the Higgs vacuum for all of these leptoquark models by studying the evolution of $\lambda _{h}$ with the energy scale. Furthermore, we investigate the stability issue following the Coleman–Weinberg effective potential approach. In Sect. 5 we describe the phenomenology of leptoquarks in light of direct and indirect bounds on their parameter space. Finally, we conclude in Sect. 6.

2 Leptoquark models

This section provides a theoretical description of the leptoquarks ${\widetilde{R}}_2$ and $\varvec{S}_3$. First, we consider the model with scalar doublet leptoquark ${\widetilde{R}}_2\,(\varvec{3,2,}1/6)$, where the numbers in brackets denote its $SU(3)_c\bigotimes SU(2)_L\bigotimes U(1)_Y$ nature. Since this leptoquark is a doublet under $SU(2)_L$, it has two components with the electromagnetic charges 2/3 and ${-1}/{3}$, and we designate them as ${\widetilde{R}}_2^{2/3}$ and ${\widetilde{R}}_2^{-1/3}$. The corresponding Lagrangian is given by:

(1)

where $D^\mu$ signifies the covariant derivative related to the kinetic term of fields, $m_2$ is the mass of the leptoquark ${\widetilde{R}}_2$ before electroweak symmetry breaking (EWSB), $\lambda _2$ and ${\widetilde{\lambda }}_2$ are the couplings for quartic interaction terms of ${\widetilde{R}}_2$ with scalar doublet field H, and the $3\times 3$ matrix $Y_2$ indicates the coupling of ${\widetilde{R}}_2$ with quarks and leptons where $L_L$ represents the SU(2) lepton doublet. For simplicity, we hide the generation index. After EWSB, the scalar field H gives rise to Higgs boson h, and the two components of ${\widetilde{R}}_2$ obtain additional contributions in their masses from the quartic coupling terms. It is important to mention that the generation indices have been suppressed here. However, to realize the full mathematical description of this model, one has to add the SM Lagrangian as well. In our notation, we denote the SM Yukawa couplings for the charged leptons, up-type quarks and down-type quarks as , and , respectively. The SM Higgs potential is given by:

$$\begin{aligned} V_0=-\,\mu _h^{} |H|^2+\lambda _h |H|^4 \quad \text {with} \quad H=\frac{1}{\sqrt{2}}\begin{pmatrix} 0\\ v+h \end{pmatrix} , \end{aligned}$$

(2)

under unitary gauge, where the tree-level mass of the Higgs boson becomes $M_h^{}=\sqrt{2\,\mu _h^{}}$, and the vacuum expectation value (VEV) of the scalar field can be expressed as $v=\sqrt{\nicefrac {\mu _h^{}}{\lambda _h}}$. After EWSB, the squared masses for the leptoquarks ${\widetilde{R}}_2^{2/3}$ and ${\widetilde{R}}_2^{1/3}$ respectively become:

$$\begin{aligned} m^2_{2,2/3}&=\ m_2^2+\frac{1}{2}\lambda _2 v^2,\qquad \qquad \nonumber \\ m^2_{2,1/3}&=\ m_2^2+\frac{1}{2}(\lambda _2+ {{\widetilde{\lambda }}}_2) v^2, \end{aligned}$$

(3)

and thus the two components of the doublet no longer remain degenerate and acquire a mass gap of $\frac{1}{2}{\widetilde{\lambda }}_2 v^2$.

In principle, there could be some other gauge-invariant dimension-four terms for ${\widetilde{R}}_2$, for example, $\epsilon _{\alpha \beta \gamma } (H^T i\sigma _2 \widetilde{R}_{2,\alpha })(\widetilde{R}_{2,\beta }^T i\sigma _2 \widetilde{R}_{2,\gamma })$ or $(\widetilde{R}_2^\dagger \widetilde{R}_2)(\widetilde{R}_2^\dagger \widetilde{R}_2)$. The first term does not conserve the baryon and lepton number separately; additionally, it initiates proton decay via the mode $p\rightarrow \pi ^+\pi ^+e^-\nu \nu$ [3, 129, 130], which in turn forces the leptoquark mass to be very high to reach the experimental value of proton lifetime. Thus, one should either neglect the term or assume that it is forbidden by some other symmetry. For example, if we impose a $Z_2^{l}$ discrete symmetry under which all the SM leptons and the leptoquarks are both odd, but other particles like quarks and the scalar doublet H are even, this particular term will be prohibited. The same effect can be achieved by imposing $Z_2^q$ discrete symmetry for which the quarks and leptoquarks are odd and all the other particles are even. On the other hand, the second term does not affect any other SM couplings up to the two-loop level. So, for simplicity, we ignore it as well.

In the second scenario, we study the extension of SM with the scalar triplet leptoquark . The three excitations of this multiplet possess the electromagnetic charges 4/3, 1/3 and ${-2}/{3}$, and therefore we designate them as $S_3^{4/3}$, $S_3^{1/3}$ and $S_3^{-2/3}$, respectively. The Lagrangian for this leptoquark is given by:

(4)

where $S_3^{ad}$ signifies $\varvec{S}_3$ in adjoint representation, $m_3$ is the mass of $\varvec{S}_3$ before EWSB, $\lambda _3$ and ${\widetilde{\lambda }}_3$ are the couplings for quartic interaction terms of this leptoquark with Higgs boson, and $Y_3$ indicates its coupling with different quarks and leptons where $Q_L$ represents the SU(2) quark doublet. It is interesting to note that the term $H^\dagger (S_3^{ad})^\dagger S_3^{ad} H$ is absent in the Lagrangian, given by Eq. (4), since it is not an independent term. It can be easily checked that $H^\dagger \,[S_3^{ad}\,(S_3^{ad})^\dagger +(S_3^{ad})^\dagger S_3^{ad}]\,H=(H^\dagger H)\, \text {Tr}[(S_3^{ad})^\dagger S_3^{ad}]$ under unitary gauge. After EWSB, the squared masses for leptoquarks $S_3^{4/3}$, $S_3^{1/3}$ and $S_3^{2/3}$ become:

$$\begin{aligned} m^2_{3,4/3}&= \ m_3^2+\frac{1}{2}\lambda _3 v^2,\qquad \nonumber \\ m^2_{3,2/3}&= \ m_3^2+\frac{1}{2}(\lambda _3+ {{\widetilde{\lambda }}}_3) v^2,\qquad \nonumber \\ m^2_{3,1/3}&=\ m_3^2+\frac{1}{2}\left( \lambda _3+\frac{1}{2} {{\widetilde{\lambda }}}_3\right) v^2, \end{aligned}$$

(5)

which lifts the degeneracy among these three states like the previous scenario. In this case, apart from the leptoquark self-quartic interactions, i.e. $\text {Tr}\big [(S_3^{ad})^\dagger S_3^{ad}(S_3^{ad})^\dagger S_3^{ad}\big ]$ and $\text {Tr}\big [(S_3^{ad})^\dagger S_3^{ad}\big ]^2$, which we neglect for simplicity as in the doublet leptoquark scenario, there could exist a diquark term like allowed by gauge symmetry. However, this term neither respects baryon and lepton number separately nor protects protons from decay through $p\rightarrow e^+\pi ^0$ or $p\rightarrow \pi ^+{{\bar{\nu }}}_e$ [3, 131,132,133]. In the same fashion as in the $\widetilde{R}_2$ case, here also one can impose $Z_2^l$ or $Z_2^q$ symmetry to forbid this term. For our analysis, we neglect it as well.

Lastly, we consider the scenario having both the leptoquarks ${\widetilde{R}}_2$ and $\varvec{S}_3$. The relevant part of the Lagrangian for this model is given by:

$$\begin{aligned} {\mathscr {L}}_{23}= {\mathscr {L}}_{2}+{\mathscr {L}}_{3} - [\kappa _h^{} H^\dagger S_3^{ad} {\widetilde{R}}_2+h.c.]. \end{aligned}$$

(6)

The interesting feature of this model is that, besides the individual interaction terms for doublet and triplet leptoquarks, it contains one additional dimension-three term which couples the doublet leptoquark to the triplet one through the Higgs boson. As in earlier cases, we have not considered the leptoquark self-quartic couplings.

In this scenario, it is important to note that although $S_3^{4/3}$ remains as mass eigenstate, the other components of $\widetilde{R}_2$ and $\varvec{S}_3$ do not. For instance, the squared mass matrix for $\widetilde{R}_2^{1/3}$ and $S_3^{1/3}$ becomes:

$$\begin{aligned} M^2_{1/3}=\begin{pmatrix} m^2_{2,1/3} &{} \frac{1}{2}\kappa _h^{} v\\ \frac{1}{2}\kappa _h^{*} v &{} m^2_{3,1/3} \end{pmatrix}, \end{aligned}$$

(7)

where $\kappa _h^{*}$ indicates the complex conjugate of $\kappa _h^{}$. Therefore, these two flavour states mix together to produce the energy eigenstates as:

$$\begin{aligned} \begin{pmatrix} \varOmega _1\\ \varOmega _2 \end{pmatrix}=\begin{pmatrix} \cos \theta _1^{}&{}\sin \theta _1^{}\, e^{i\phi _1}\\ -\sin \theta _1^{} \, e^{-i\phi _1}&{} \cos \theta _1^{} \end{pmatrix} \begin{pmatrix} \widetilde{R}_2^{1/3}\\ S_3^{1/3} \end{pmatrix}, \end{aligned}$$

(8)

where the mixing angle $\theta _1^{}$ and the CP-violating phase $\phi _1$ are given by:

$$\begin{aligned} \tan 2\theta _1^{}=-\,\bigg (\frac{v\;|\kappa _h|}{m_{3,1/3}^2-m_{2,1/3}^2}\bigg ) \quad \text {and} \quad e^{i\phi _1}=\frac{\kappa _h^*}{|\kappa _h|}. \end{aligned}$$

(9)

The squared masses for the energy eigenstates $\varOmega _{1,2}$ are given by:

$$\begin{aligned} m^2(\varOmega _{1,2})&= \frac{1}{2}\Bigg [\big (m_{3,1/3}^2+m_{2,1/3}^2\big ) \nonumber \\&\quad \mp \sqrt{\big (m_{3,1/3}^2-m_{2,1/3}^2\big )^2+v^2\,|\kappa _h|^2}\;\Bigg ]. \end{aligned}$$

(10)

Similarly, the squared mass matrix for $\widetilde{R}_2^{2/3}$ and $S_3^{2/3}$ becomes:

$$\begin{aligned} M^2_{2/3}=\begin{pmatrix} m^2_{2,1/3} &{} -\frac{1}{\sqrt{2}}\kappa _h^{} v\\ -\frac{1}{\sqrt{2}}\kappa _h^{*} v &{} m^2_{3,1/3} \end{pmatrix}, \end{aligned}$$

(11)

and these two flavour states also mix together to produce the energy eigenstates as:

$$\begin{aligned} \begin{pmatrix} \chi _1\\ \chi _2 \end{pmatrix}=\begin{pmatrix} \cos \theta _2^{}&{}\sin \theta _2^{}\, e^{i\phi _2}\\ -\sin \theta _2^{}\, e^{-i\phi _2}&{} \cos \theta _2^{} \end{pmatrix} \begin{pmatrix} \widetilde{R}_2^{2/3}\\ S_3^{2/3} \end{pmatrix}, \end{aligned}$$

(12)

where the mixing angle $\theta _2^{}$ and the CP-violating phase $\phi _2$ are given by:

$$\begin{aligned} \tan 2\theta _2^{}=\bigg (\frac{\sqrt{2} \,v\;|\kappa _h|}{m_{3,2/3}^2-m_{2,2/3}^2}\bigg ) \text{ and } e^{i\phi _2}=\frac{\kappa _h^*}{|\kappa _h|}. \end{aligned}$$

(13)

The squared masses for the energy eigenstates $\varOmega _{1,2}$ are given by:

$$\begin{aligned} m^2(\chi _{1,2})&=\ \frac{1}{2}\Bigg [\big (m_{3,2/3}^2+m_{2,2/3}^2\big ) \nonumber \\&\quad \mp \sqrt{\big (m_{3,2/3}^2-m_{2,2/3}^2\big )^2+2\,v^2\,|\kappa _h|^2}\;\Bigg ]. \end{aligned}$$

(14)

As a special case, if $\kappa _h$ becomes zero, i.e. no mixing between doublet and triplet, then the mass and flavour states remain the same, i.e. the mixing angle becomes zero. On the other hand, if the masses of the doublet and triplet flavour eigenstates become the same, the mixing angles turn to $\pi /4$ and mass differences of $v|\kappa _h|$ and $\sqrt{2} v|\kappa _h|$ are generated between the mass-eigenstates with charge 1/3 and 2/3, respectively.

Now, regarding the generation of leptoquarks, we follow two different conventions: (a) there is one generation of leptoquark that couples to one generation of quark and lepton only; (b) there exist three generations of leptoquarks, each of which couples to one generation of quark and lepton only. Both conventions have different pros and cons when considering several low-energy and collider bounds on leptoquarks. However, for our analysis we study both of them. For the first scenario, we consider only diagonal coupling of the leptoquarks given by $Y_\gamma ^{rs}=Y_\phi \,\text {diag}(1,0,0)$, with r, s being the generation indices for quarks and leptons and $\gamma \in \{2,3\}$. Obviously, one can choose $\text {diag}(0,1,0)$ or $\text {diag}(0,0,1)$ as well. In the second case, we assume $Y_{\gamma ,i}^{rs}=Y_\phi \,\delta ^{ir}\delta ^{is}$, with i being the generation of the leptoquark. In this scenario, the terms $\lambda _\gamma$ and ${\widetilde{\lambda }}_\gamma$ also become $3\times 3$ matrices, but we also consider them diagonal, restricting the generation mixing of the leptoquarks.

3 Perturbativity

In this section we study the perturbativity of the theory with respect to different dimensionless couplings. It is well known that expansion of amplitude or cross section in perturbative series is plausible only when the expansion parameter is less than unity. Therefore, the constraints that must be satisfied by different couplings in order to respect the perturbativity of the theory are as follows: [118, 120, 121]:

$$\begin{aligned} |\lambda _\alpha |\le 4\pi , \quad |{{\widetilde{\lambda }}}_\gamma |\le 4\pi , \quad |g_k|\le 4\pi \quad |Y_l^{rs}|\le \sqrt{4\pi }, \end{aligned}$$

(15)

where $\lambda _\alpha$ and ${\widetilde{\lambda }}_\gamma$, with $\alpha \in \{h,2,3\}$ and $\gamma \in \{2,3\}$, indicate the quartic couplings of the Higgs boson with leptoquarks as well as the self-quartic coupling of the Higgs boson, $g_k$ with $k\in \{1,2,3\}$ signifies the gauge couplings corresponding to $U(1)_Y$, $SU(2)_L$ and $SU(3)_C$ gauge symmetry, respectively, and $Y_l^{rs}$ with represents the (r, s) element of the Yukawa (or Yukawa-like) coupling matrices for quarks and leptons. We generate two-loop beta functions for different couplings through SARAH [134, 135] in the $\overline{\text {MS}}$ scheme and analyse them. We use the usual definition of beta function

$$\begin{aligned} \beta (x)=\frac{\partial x}{\partial (\log \mu )}, \end{aligned}$$

(16)

when considering the running of any coupling parameter x with the energy scale $\mu$. The running of different parameters in generalized field theory with dimensional regularization [136] in the $\overline{\text {MS}}$ scheme has already been addressed in Refs. [137,138,139,140]. The RG evolution of various parameters under the SM has been discussed in [141,142,143,144].

3.1 Gauge couplings

We first discuss the renormalization group (RG) evolution of the gauge couplings. Since the doublet and triplet leptoquarks posses all three gauge charges, namely weak hypercharge, isospin and colour, the running of all gauge couplings will differ from the SM. However, in some scenarios the weak coupling constant $g_2$ gradually increases to hit the Landau pole at some energy scale, which eventually leads to sudden divergences in the other two gauge couplings. Therefore, we present the running of $g_2$ at the beginning.

3.1.1 Beta function of $g_2$: a brief review

It is well established that for any non-Abelian gauge group $G=SU(N)$, the one-loop beta function of the gauge coupling g is given by:

$$\begin{aligned} \beta (g)^{1\mathrm{-loop}}=\frac{g^3}{16\pi ^2}\,\bigg [\frac{4}{3}n_f\,T(R_f)+\frac{1}{3}n_s\,T(R_s)-\frac{11}{3}C_2(G)\bigg ]\nonumber \\ \end{aligned}$$

(17)

where $n_f$ is the number of Dirac fermionic multiplets in representation $R_f$, $n_s$ is the number of complex scalar multiplets in representation $R_s$, $C_2(G)$ is the quadratic Casimir of the gauge group G and is equal to N since the gauge fields lie in the adjoint representation of G, and finally, $T(R_{f/s})$ are other Casimir invariants defined by , with being the generator of the Lie algebra in the representation $R_{f/s}$. At this point, it is worth mentioning that one should replace the factor 4/3 with 2/3 in Eq. (17) when dealing with Weyl or Majorana fermions, and similarly, the factor 1/3 must be replaced with 1/6 for real scalar multiplets.

If we consider the one-loop beta function of weak coupling constant $g_2$ in the SM, the corresponding gauge group will be $SU(2)_L$. Hence, the fermionic contribution would come from 12 Weyl fermionic doublets: (a) three generations of leptonic doublets and (b) nine quark doublets (three generations and three colours). However, since they are all Weyl fermions due to the left chiral nature of the weak interaction, one must take the 2/3 factor instead of 4/3 as the coefficient of the term $n_f\,T(R_f)$ in Eq. (17). On the other hand, there is only one charged scalar doublet interacting weakly in the SM. Moreover, $T(R_{f/s})=1/2$ for all the fermions and scalar under $SU(2)_L$ gauge group, as they are all in fundamental representation. Thus one-loop beta function of $g_2$ in the SM becomes:

$$\begin{aligned} \beta (g_2)^{1\mathrm{-loop}}_{\mathrm{SM}}&= \frac{g^3_2}{16\pi ^2}\,\bigg [\bigg (\frac{2}{3}\times 12\times \frac{1}{2}\bigg )+\bigg (\frac{1}{3}\times 1\times \frac{1}{2}\bigg ) \nonumber \\&\quad - \bigg (\frac{11}{3}\times 2\bigg )\bigg ]=-\,\frac{19}{6}\,\bigg (\frac{g^3_2}{16\pi ^2}\bigg ). \end{aligned}$$

(18)

Now, if we add one generation of scalar doublet leptoquark ${\widetilde{R}}_2$ to the SM, we can express the one-loop beta function of $g_2$ as:

$$\begin{aligned} \beta (g_2)^{1\mathrm{-loop}}_{{\widetilde{R}}_2,1\mathrm{-gen}} = \beta (g_2)^{1\mathrm{-loop}}_{\mathrm{SM}} + \varDelta \beta (g_2)^{1\mathrm{-loop}}_{{\widetilde{R}}_2} , \end{aligned}$$

(19)

where the term $\varDelta \beta (g_2)^{1\mathrm{-loop}}_{{\widetilde{R}}_2}$ signifies the sole contribution from the single generation of the leptoquark ${\widetilde{R}}_2$. Since ${\widetilde{R}}_2$ is a complex scalar in fundamental representation of $SU(2)_L$ having three colour choices, we find:

$$\begin{aligned}&\varDelta \beta (g_2)^{1\mathrm{-loop}}_{{\widetilde{R}}_2}= \frac{g^3_2}{16\pi ^2}\,\bigg (\frac{1}{3}\times 3\times \frac{1}{2}\bigg )=\frac{1}{2}\,\bigg (\frac{g^3_2}{16\pi ^2}\bigg ),\nonumber \\&\quad \implies \beta (g_2)^{1\mathrm{-loop}}_{{\widetilde{R}}_2,1\mathrm{-gen}} = -\,\frac{8}{3}\,\bigg (\frac{g^3_2}{16\pi ^2}\bigg ). \end{aligned}$$

(20)

Consequently, for the extension of SM with three generations of ${\widetilde{R}}_2$, the one-loop beta function of $g_2$ becomes:

$$\begin{aligned} \beta (g_2)^{1\mathrm{-loop}}_{{\widetilde{R}}_2,3\mathrm{-gen}}&= \ \beta (g_2)^{1\mathrm{-loop}}_{\mathrm{SM}} + 3\,\varDelta \beta (g_2)^{1\mathrm{-loop}}_{{\widetilde{R}}_2} \nonumber \\&=-\,\frac{5}{3}\,\bigg (\frac{g^3_2}{16\pi ^2}\bigg ). \end{aligned}$$

(21)

Similarly, the one-loop beta function for $g_2$ in the SM plus one generation of scalar triplet leptoquark $\varvec{S}_3$ can also be expressed as:

$$\begin{aligned} \beta (g_2)^{1\mathrm{-loop}}_{\varvec{S}_3,1\mathrm{-gen}} = \beta (g_2)^{1\mathrm{-loop}}_{\mathrm{SM}} + \varDelta \beta (g_2)^{1\mathrm{-loop}}_{\varvec{S}_3} , \end{aligned}$$

(22)

where $\varDelta \beta (g_2)^{1\mathrm{-loop}}_{\varvec{S}_3}$ contains the sole contribution of $\varvec{S}_3$ with one generation. However, since $\varvec{S}_3$ is a complex scalar triplet under $SU(2)_L$, it will be in adjoint representation; hence, $T(R_{\varvec{S}_3})=2$.²

Furthermore, there will be three copies of $\varvec{S}_3$ depending on the colour charges. Thus, one finds the contribution of $\varvec{S}_3$ in the one-loop beta function of $g_2$ as:

$$\begin{aligned}&\varDelta \beta (g_2)^{1\mathrm{-loop}}_{\varvec{S}_3}\ = \frac{g^3_2}{16\pi ^2}\,\bigg (\frac{1}{3}\times 3\times 2\bigg )=2\,\bigg (\frac{g^3_2}{16\pi ^2}\bigg ), \nonumber \\&\quad \implies \beta (g_2)^{1\mathrm{-loop}}_{\varvec{S}_3,1\mathrm{-gen}} = -\,\frac{7}{6}\,\bigg (\frac{g^3_2}{16\pi ^2}\bigg ), \end{aligned}$$

(23)

and the one-loop beta function of $g_2$ with SM plus three generations of $\varvec{S}_3$ necessarily becomes:

$$\begin{aligned} \beta (g_2)^{1\mathrm{-loop}}_{\varvec{S}_3,3\mathrm{-gen}}= & {} \beta (g_2)^{1\mathrm{-loop}}_{\mathrm{SM}} + 3\, \varDelta \beta (g_2)^{1\mathrm{-loop}}_{\varvec{S}_3} \nonumber \\= & {} \frac{17}{6}\,\bigg (\frac{g^3_2}{16\pi ^2}\bigg ). \end{aligned}$$

(24)

If the SM is extended with both $\widetilde{R}_2$ and $\varvec{S}_3$, the one-loop beta function of $g_2$ can be calculated as:

$$\begin{aligned}&\beta (g_2)^{1\mathrm{-loop}}_{{\widetilde{R}}_2+\varvec{S}_3,1\mathrm{-gen}} = \beta (g_2)^{1\mathrm{-loop}}_{\mathrm{SM}} + \varDelta \beta (g_2)^{1\mathrm{-loop}}_{{\widetilde{R}}_2} \nonumber \\&\quad + \varDelta \beta (g_2)^{1\mathrm{-loop}}_{\varvec{S}_3}=-\,\frac{2}{3}\,\bigg (\frac{g^3_2}{16\pi ^2}\bigg ),&\end{aligned}$$

(25)

$$\begin{aligned}&\beta (g_2)^{1\mathrm{-loop}}_{{\widetilde{R}}_2+\varvec{S}_3,3\mathrm{-gen}} = \beta (g_2)^{1\mathrm{-loop}}_{\mathrm{SM}} + 3\,\varDelta \beta (g_2)^{1\mathrm{-loop}}_{{\widetilde{R}}_2}\nonumber \\&\quad + 3\,\varDelta \beta (g_2)^{1\mathrm{-loop}}_{\varvec{S}_3}=\frac{13}{3}\,\bigg (\frac{g^3_2}{16\pi ^2}\bigg ).&\end{aligned}$$

(26)

Now, we use SARAH to generate the two-loop contributions. For convenience, we define:

(27)

where . Thus the beta function of $g_2$ up to two-loop order for the different models we are working with becomes:

(28)

(29)

(30)

(31)

(32)

(33)

(34)

Table 1 Initial values for different SM parameters required for RG evolution at electroweak (EW) scale

In Eqs. (30), (32) and (34), the index i represents the generation of the leptoquark. Now, as we have defined $\varDelta \beta (g_2)$ for one loop in Eqs. (19) and (22), one can define it for two loops in similar fashion. Then one can easily verify that the above-described two-loop beta functions obey the following relations:

(35)

(36)

(37)

(38)

where for any parameter indicates the beta function of that parameter with the replacement of to with $\gamma \in \{2,3\}$ and i representing the generation. Similar notation is applicable for $\varDelta \beta$.

3.1.2 Scale variation of $g_2$

Fig. 1

Running of gauge coupling $g_2$ with the energy scale $\mu$ for SM and different leptoquark scenarios with one loop and two loops. The green curve represents the SM; the red and yellow curves signify the extension of the SM with one and three generations of $\widetilde{R}_2$, respectively; the addition of one and three generations of $\varvec{S}_3$ to SM are indicated by the blue and cyan lines, respectively; the brown and dashed black curves illustrate the SM extension with one and three generations of both $\widetilde{R}_2$ and $\varvec{S}_3$, respectively. The initial values for the SM parameters are listed in Table 1 along with $Y_\phi =0.1$

Using the above Eqs.³ (28)–(34), we plot the dependence of coupling $g_2$ for different models on the energy scale $\mu$ in Fig. 1. While Fig. 1a depicts the behaviour of $g_2$ at one loop, Fig. 1b illustrates the same for the two-loop level. The SM is represented by the green curve; the red and yellow lines depict the extension of the SM with one and three generations of $\widetilde{R}_2$, respectively; the blue and cyan lines indicate the addition of one and three generations of $\varvec{S}_3$, respectively, to SM; finally, the brown and dashed black curves illustrate the SM extension with one and three generations of both doublet and triplet leptoquarks, respectively. The initial two-loop values at the electroweak (EW) scale for gauge couplings $g_1, g_2, g_3$, Higgs quartic coupling $\lambda _h$ and top-quark Yukawa coupling are given in Table 1, with the contributions from other Yukawa couplings neglected [143, 144]. Although the plots are made assuming $Y_\phi$ to be 0.1, they do not change significantly with the alteration of $Y_\phi$, since the dominant contribution in the two-loop beta function of $g_2$ comes from different gauge couplings, as can be realized from Eqs. (28)–(34). This statement holds for other gauge couplings as well.

In Fig. 1a, b, we present the variation in $g_2$ with respect to the scale $\mu$ at the one- and two-loop levels for the above-mentioned leptoquark scenarios. The ordering of different curves in Fig. 1 is mainly controlled by the one-loop beta functions for different models which are presented in Eqs. (18)–(26). The one-loop beta function of $g_2$ under SM is $-\frac{19}{6}(\frac{g_2^3}{16\pi ^2})$, which is enhanced to $-\frac{8}{3}(\frac{g_2^3}{16\pi ^2})$ and $-\frac{5}{3}(\frac{g_2^3}{16\pi ^2})$ for one and three generations of $\widetilde{R}_2$, respectively. However, due to the greater number of components of leptoquarks in $\varvec{S}_3$, the positive contributions will be greater. The one-loop beta functions of $g_2$ for $\varvec{S}_3$ with one and three generations become $-\frac{7}{6}(\frac{g_2^3}{16\pi ^2})$ and $\frac{17}{6}(\frac{g_2^3}{16\pi ^2})$. For the combined scenario of these two leptoquarks, the positive effects are even stronger. For one and three generations of the combined case, the beta functions become $-\frac{2}{3}(\frac{g_2^3}{16\pi ^2})$ and $\frac{13}{3}(\frac{g_2^3}{16\pi ^2})$, respectively. It is interesting to note that except for three generations of $\varvec{S}_3$ (cyan line) and $\widetilde{R}_2+\varvec{S}_3$ (black dotted curve), coupling, $g_2$ decreases monotonically for all scenarios due to the negative sign in the one-loop beta function, ensuring asymptotic freedom of weak interaction. However, when considering one-loop effects only, Planck scale perturbativity is achieved in all the scenarios, since the Landau pole in the two above-mentioned cases appears beyond the Plank scale, as can be seen from Fig. 1a. On the other hand, due to the positive value of the one-loop beta function, the gauge coupling $g_2$ in the three generations of the $\varvec{S}_3$ (cyan) and $\widetilde{R}_2+\varvec{S}_3$ (black dashed) models increases monotonically. Now, for the two-loop case, all the models acquire additional positive effects that push the RG evolution curves upwards. Therefore, the two-loop beta functions of the $\varvec{S}_3$ and $\widetilde{R}_2+\varvec{S}_3$ models hit the Landau pole at lower scales, i.e. $10^{19.7}$ GeV (just above the Planck scale) and $10^{14.4}$ GeV (below the GUT scale), respectively, as can be observed from Fig. 1(c).

Fig. 2

Running of gauge coupling $g_3$ with the energy scale $\mu$ for different leptoquark models at one-loop and two-loop order. The green curve represents SM; the red and yellow ones signify the extension of SM with one and three generations of $\widetilde{R}_2$, respectively; the addition of one and three generations of $\varvec{S}_3$ to SM is indicated by the blue and cyan lines, respectively; the brown and dashed black curves illustrate the SM extension with one and three generations of both $\widetilde{R}_2$ and $\varvec{S}_3$, respectively. The initial values for SM parameters are listed in Table 1 along with $Y_\phi =0.1$

3.1.3 Beta function of $g_3$: a brief review

In the case of the SM, as the scalar and leptons are colour neutral, the one-loop beta function of strong coupling $g_3$ receives contributions only from six quarks which are essentially Dirac fermionic colour triplets under the SU(3) gauge group, Thus, substituting $T(R_f)=1/2$ and $C_2(G)=3$ in Eq. (17), we get:

$$\begin{aligned} \beta (g_3)^{1\mathrm{-loop}}_{\mathrm{SM}}= & {} \frac{g^3_3}{16\pi ^2}\,\bigg [\bigg (\frac{4}{3}\times 6\times \frac{1}{2}\bigg ) - \bigg (\frac{11}{3}\times 3\bigg )\bigg ]\nonumber \\= & {} -\,7\,\bigg (\frac{g^3_3}{16\pi ^2}\bigg ). \end{aligned}$$

(39)

Now, all the leptoquarks are colour triplet complex scalars, i.e. they are in fundamental representation of SU(3) enforcing $T(R_s)=1/2$. However, for the doublet leptoquark we have two such copies of triplets, namely $\widetilde{R}_2^{2/3}$ and $\widetilde{R}_2^{1/3}$, whereas there are three scalar triplets for $\varvec{S}_3$, namely $S_3^{4/3}$, $S_3^{2/3}$ and $S_3^{1/3}$. Thus the sole contribution of one generation $\widetilde{R}_2^{2/3}$ and $\varvec{S}_3$ in the beta function of $g_3$, as described in the previous subsection for $g_2$, can be written as:

$$\begin{aligned} \varDelta \beta (g_3)^{1\mathrm{-loop}}_{\widetilde{R}_2}=\frac{g^3_3}{16\pi ^2}\,\bigg (\frac{1}{3}\times 2\times \frac{1}{2}\bigg )=\frac{1}{3}\,\bigg (\frac{g^3_3}{16\pi ^2}\bigg ), \end{aligned}$$

(40)

$$\begin{aligned} \varDelta \beta (g_3)^{1\mathrm{-loop}}_{\varvec{S}_3}=\frac{g^3_3}{16\pi ^2}\,\bigg (\frac{1}{3}\times 3\times \frac{1}{2}\bigg )=\frac{1}{2}\,\bigg (\frac{g^3_3}{16\pi ^2}\bigg ). \end{aligned}$$

(41)

Therefore, the one-loop beta functions of strong coupling $g_3$ for different SM extensions with $\widetilde{R}_2$ and $\varvec{S}_3$ are as follows:

$$\begin{aligned}&\beta (g_3)^{1\mathrm{-loop}}_{{\widetilde{R}}_2,1\mathrm{-gen}} = \beta (g_3)^{1\mathrm{-loop}}_{\mathrm{SM}} + \varDelta \beta (g_3)^{1\mathrm{-loop}}_{{\widetilde{R}}_2}\nonumber \\&\quad =-\,\frac{20}{3}\,\bigg (\frac{g^3_3}{16\pi ^2}\bigg ), \end{aligned}$$

(42)

$$\begin{aligned}&\beta (g_3)^{1\mathrm{-loop}}_{{\widetilde{R}}_2,3\mathrm{-gen}} = \beta (g_3)^{1\mathrm{-loop}}_{\mathrm{SM}} + 3\,\varDelta \beta (g_3)^{1\mathrm{-loop}}_{{\widetilde{R}}_2}\nonumber \\&\quad =-\,6\,\bigg (\frac{g^3_3}{16\pi ^2}\bigg ), \end{aligned}$$

(43)

$$\begin{aligned}&\beta (g_3)^{1\mathrm{-loop}}_{\varvec{S}_3,1\mathrm{-gen}} = \beta (g_3)^{1\mathrm{-loop}}_{\mathrm{SM}} + \varDelta \beta (g_3)^{1\mathrm{-loop}}_{\varvec{S}_3}\nonumber \\&\quad =-\,\frac{13}{2}\,\bigg (\frac{g^3_3}{16\pi ^2}\bigg ), \end{aligned}$$

(44)

$$\begin{aligned}&\beta (g_3)^{1\mathrm{-loop}}_{\varvec{S}_3,3\mathrm{-gen}} = \beta (g_3)^{1\mathrm{-loop}}_{\mathrm{SM}} + 3\,\varDelta \beta (g_3)^{1\mathrm{-loop}}_{\varvec{S}_3}\nonumber \\&\quad =-\,\frac{11}{2}\,\bigg (\frac{g^3_3}{16\pi ^2}\bigg ), \end{aligned}$$

(45)

$$\begin{aligned}&\beta (g_3)^{1\mathrm{-loop}}_{{\widetilde{R}}_2+\varvec{S}_3,1\mathrm{-gen}} = \beta (g_3)^{1\mathrm{-loop}}_{\mathrm{SM}} + \varDelta \beta (g_3)^{1\mathrm{-loop}}_{{\widetilde{R}}_2}\nonumber \\&\quad + \varDelta \beta (g_3)^{1\mathrm{-loop}}_{\varvec{S}_3}=-\,\frac{37}{6}\,\bigg (\frac{g^3_3}{16\pi ^2}\bigg ), \end{aligned}$$

(46)

$$\begin{aligned}&\beta (g_3)^{1\mathrm{-loop}}_{{\widetilde{R}}_2+\varvec{S}_3,3\mathrm{-gen}} = \beta (g_3)^{1\mathrm{-loop}}_{\mathrm{SM}} + 3\,\varDelta \beta (g_3)^{1\mathrm{-loop}}_{{\widetilde{R}}_2}\nonumber \\&\qquad + 3\,\varDelta \beta (g_3)^{1\mathrm{-loop}}_{\varvec{S}_3}=-\,\frac{9}{2}\,\bigg (\frac{g^3_3}{16\pi ^2}\bigg ). \end{aligned}$$

(47)

The two-loop beta functions of $g_3$ for all these models are listed in Appendix A.

3.1.4 Scale variation of $g_3$

The variations in strong gauge coupling $g_3$ at the one-loop and two-loop levels with energy scale $\mu$ for different models are depicted in Fig. 2. The left panel shows the one-loop results, while the right panel indicates the full two-loop contributions. The same colour code as mentioned in previous section is also followed here. The relative positions of the different curves in this plot are mainly determined by coefficients of $(\frac{g_3^3}{16\pi ^2})$ in the one-loop beta functions, given by Eqs. (39)–(47). This coefficient for SM (green) is $-7$, which is enhanced to $-20/3$, $-13/2$ and $-37/6$ for the $\widetilde{R}_2$ (red), $\varvec{S}_3$ (blue) and the combined scenario (brown) for one generation, respectively. For the case of three generations, this factor receives even more contributions to become $-6$, $-11/2$ and $-9/2$, respectively, for the $\widetilde{R}_2$ (yellow) , $\varvec{S}_3$ (cyan) and the combined scenario (black dashed). As the one-loop beta function of $g_3$ for all the models remains negative, $g_3$ decreases gradually with the increase in energy, showing asymptotic freedom. As can be seen from Fig. 2a, all the models show Planck scale perturbativity at one-loop order. At two-loop order, all of these curves shift upwards due to the additional positive contributions as shown in Appendix A. All but two of the models exhibit no unusual behaviour. However, as $g_2$ hits the Landau pole at $10^{19.7}$ GeV and $10^{14.4}$ GeV for three generations of the $\varvec{S}_3$ (cyan) and $\widetilde{R}_2+\varvec{S}_3$ (black dashed) models, respectively, $g_3$ shows sudden divergence for these two models at the above-mentioned energy scales (see Fig. 2b).

3.1.5 Beta function of $g_1$: a brief review

The one-loop beta function for gauge coupling $g_1$ is given by:

(48)

where signify the hypercharge of the Weyl fermions and the scalars, respectively.^4,5 The 3/5 factor arises because of SU(5) normalization of the coupling $g_1$. On the other hand, since the gauge boson interacts with left- and right-handed fermions with different hypercharges, one has to sum over the contributions from all the Weyl fermions, and hence the 2/3 factor appears for the fermionic effects instead of 4/3. In the SM, there are 18 left-handed quarks (six flavours, three colours) with hypercharge 1/6, nine right-handed up-type quarks (three generations, three colours) with hypercharge 2/3, nine right-handed down-type quarks (three generations, three colours) with hypercharge $-1/3$, six left-handed leptons with and three right-handed charged leptons with . Additionally, there are two scalars ($H^+$ and $H^0$, the components of scalar doublet H), each with . Thus the one-loop beta function for $g_1$ in the SM becomes:

$$\begin{aligned} \beta (g_1)^{1\mathrm{-loop}}_{\mathrm{SM}}&=\frac{3}{5}\,\bigg (\frac{g_1^3}{16\pi ^2}\bigg ) \,\bigg [\,\frac{2}{3}\times \bigg (\frac{1}{2}+4+1+\frac{3}{2}+3\bigg )\nonumber \\&\quad +\frac{1}{3}\times \frac{1}{2}\bigg ]=\frac{41}{10}\,\bigg (\frac{g_1^3}{16\pi ^2}\bigg ). \end{aligned}$$

(49)

Fig. 3

Running of gauge coupling $g_1$ with the energy scale $\mu$ for different leptoquark models at one-loop and two-loop order. The green curve represents SM; the red and yellow ones signify extension of SM with one and three generations of $\widetilde{R}_2$, respectively; addition of one and three generations of $\varvec{S}_3$ to SM are indicated by the blue and cyan lines, respectively; the brown and dashed black curves illustrate the SM extension with one and three generations of both $\widetilde{R}_2$ and $\varvec{S}_3$, respectively. The initial values for SM parameters are listed in Table 1 along with $Y_\phi =0.1$

Now, for SM plus one generation of $\widetilde{R}_2$, contribution from six scalars (two flavours, three colours) with hypercharge 1/6 needs to be added to the SM contribution. Similarly, the effects of nine scalars (three flavours, three colours) with must be considered when dealing with SM extension by one generation of $\varvec{S}_3$. Thus the sole contribution from one generation of $\widetilde{R}_2$ and $\varvec{S}_3$ to the one-loop beta function of $g_1$ can be calculated as:

$$\begin{aligned}&\varDelta \beta (g_1)^{1\mathrm{-loop}}_{\widetilde{R}_2}=\frac{3}{5}\,\bigg (\frac{g_1^3}{16\pi ^2}\bigg )\,\bigg [\frac{1}{3}\times 6 \times \frac{1}{36}\bigg ]=\frac{1}{30}\,\bigg (\frac{g_1^3}{16\pi ^2}\bigg ), \end{aligned}$$

(50)

$$\begin{aligned}&\varDelta \beta (g_1)^{1\mathrm{-loop}}_{\varvec{S}_3}=\frac{3}{5}\,\bigg (\frac{g_1^3}{16\pi ^2}\bigg )\,\bigg [\frac{1}{3}\times 9 \times \frac{1}{9}\bigg ]=\frac{1}{5}\,\bigg (\frac{g_1^3}{16\pi ^2}\bigg ), \end{aligned}$$

(51)

and hence, the one-loop beta function of $g_1$ for different models we considered becomes:

$$\begin{aligned}&\beta (g_1)^{1\mathrm{-loop}}_{{\widetilde{R}}_2,1\mathrm{-gen}} = \beta (g_1)^{1\mathrm{-loop}}_{\mathrm{SM}} + \varDelta \beta (g_1)^{1\mathrm{-loop}}_{{\widetilde{R}}_2}\nonumber \\&\quad =\frac{62}{15}\,\bigg (\frac{g^3_1}{16\pi ^2}\bigg ), \end{aligned}$$

(52)

$$\begin{aligned}&\beta (g_1)^{1\mathrm{-loop}}_{{\widetilde{R}}_2,3\mathrm{-gen}} = \beta (g_1)^{1\mathrm{-loop}}_{\mathrm{SM}} + 3\,\varDelta \beta (g_1)^{1\mathrm{-loop}}_{{\widetilde{R}}_2}\nonumber \\&\quad =\frac{21}{5}\,\bigg (\frac{g^3_1}{16\pi ^2}\bigg ), \end{aligned}$$

(53)

$$\begin{aligned}&\beta (g_1)^{1\mathrm{-loop}}_{\varvec{S}_3,1\mathrm{-gen}} = \beta (g_1)^{1\mathrm{-loop}}_{\mathrm{SM}} + \varDelta \beta (g_1)^{1\mathrm{-loop}}_{\varvec{S}_3}\nonumber \\&\quad =\frac{43}{10}\,\bigg (\frac{g^3_1}{16\pi ^2}\bigg ), \end{aligned}$$

(54)

$$\begin{aligned}&\beta (g_1)^{1\mathrm{-loop}}_{\varvec{S}_3,3\mathrm{-gen}} = \beta (g_1)^{1\mathrm{-loop}}_{\mathrm{SM}} + 3\,\varDelta \beta (g_1)^{1\mathrm{-loop}}_{\varvec{S}_3}\nonumber \\&\quad =\frac{47}{10}\,\bigg (\frac{g^3_1}{16\pi ^2}\bigg ), \end{aligned}$$

(55)

$$\begin{aligned}&\beta (g_1)^{1\mathrm{-loop}}_{{\widetilde{R}}_2+\varvec{S}_3,1\mathrm{-gen}} = \beta (g_1)^{1\mathrm{-loop}}_{\mathrm{SM}} + \varDelta \beta (g_1)^{1\mathrm{-loop}}_{{\widetilde{R}}_2}\nonumber \\&\quad + \varDelta \beta (g_1)^{1\mathrm{-loop}}_{\varvec{S}_3}=\frac{13}{3}\,\bigg (\frac{g^3_1}{16\pi ^2}\bigg ), \end{aligned}$$

(56)

$$\begin{aligned}&\beta (g_1)^{1\mathrm{-loop}}_{{\widetilde{R}}_2+\varvec{S}_3,3\mathrm{-gen}} = \beta (g_1)^{1\mathrm{-loop}}_{\mathrm{SM}} + 3\,\varDelta \beta (g_1)^{1\mathrm{-loop}}_{{\widetilde{R}}_2}\nonumber \\&\qquad + 3\,\varDelta \beta (g_1)^{1\mathrm{-loop}}_{\varvec{S}_3}=\frac{24}{5}\,\bigg (\frac{g^3_1}{16\pi ^2}\bigg ). \end{aligned}$$

(57)

The two-loop beta functions of $g_1$ for all these models are listed in Appendix B.

3.1.6 Scale variation of $g_1$

The variations in $g_1$ with the energy scale $\mu$ are displayed in the Fig. 3. The left panel illustrates the one-loop effects and the right panel demonstrates the two-loop effects. The colour codes are same as mentioned before. Likewise, the positions of different curves are mainly controlled by the one-loop beta functions, given by Eqs. (49)–(57). The coefficient of $(g_1^3/16\pi ^2)$ in the SM scenario is 41/10, which is enhanced to 62/15 and 21/5 for one (red) and three (yellow) generations of $\widetilde{R}_2$, respectively. For $\varvec{S}_3$ with one (blue) and three (cyan) generations, this factor increases to 43/10 and 47/10, respectively. For the combined scenario with one (brown) and three generations (black dashed), this prefactor of $(g_1^3/16\pi ^2)$ in the one-loop beta function of $g_1$ becomes 13/3 and 24/5, respectively. Since the one-loop beta functions for all the models are positive, $g_1$ increases moderately with energy. There is also no divergence for one-loop running of $g_1$ in any of the models up to the Planck scale, which can be verified from Fig. 3a. Furthermore, the two-loop beta function of $g_1$ receives additional positive contributions, presented in Appendix B, which moves all the curves of Fig. 3a in a slightly upward direction, resulting in Fig. 3b. Although all the other scenarios behave smoothly when taking into account the two-loop corrections, $g_1$ for three generations of $\varvec{S}_3$ (cyan) and $\widetilde{R}_2+\varvec{S}_3$ (black dashed) models goes to infinity abruptly at $10^{19.7}$ GeV and $10^{14.4}$ GeV, respectively, due to the divergence of $g_2$.

Thus, we find that the running of gauge couplings at two-loop order for different leptoquark models is predominantly regulated by the corresponding one-loop beta functions, which are entirely reliant on the properties of the gauge group and the number of different types of particles existing in the model. The two-loop corrections insert additional positive contributions to the running of the gauge couplings. The Yukawa couplings of the SM as well as of leptoquarks affect the RG evolution of gauge couplings only at two-loop order, and therefore with the changes in Yukawa couplings of leptoquarks, we do not observe any significant changes. However, it is interesting to note that Higgs-leptoquark quartic couplings do not appear explicitly in the two-loop beta functions of the gauge couplings at all. It is worth mentioning again that the demand of Planck scale perturbativity rules out the three generations of the $\widetilde{R}_2+\varvec{S}_3$ scenario due to the appearance of divergences at much lower scale in the two-loop running of the gauge coupling $g_2$. On the other hand, the model with three generations of $\varvec{S}_3$ is marginally allowed from Planck scale stability since the gauge coupling $g_2$ hits the Landau pole at slightly higher energy scale. These divergences force the other gauge couplings as well as the Yukawa couplings of the top quark and leptoquarks (see Appendix C and Appendix D) for these models to diverge at the two-loop level.

3.2 Higgs-leptoquark quartic couplings

Now, we step forward to investigate the perturbative bounds on Higgs-leptoquark quartic couplings.

3.2.1 Perturbativity of ${\widetilde{R}}_2$

Fig. 4

Variation in leptoquark-Higgs quartic coupling $\lambda _2$ and ${\widetilde{\lambda }}_2$ with perturbative scale for the doublet leptoquark ${\widetilde{R}}_2$ with three generations. For plots in the first row, $\lambda _2$ variation is considered for any one generation of the leptoquark, and other generations are defined as $\lambda _2^{jj}$. The other quartic couplings are designated by ${\widetilde{\lambda }}_2^{jj}$ for all three generations of leptoquark. The variations are taken for four different EW values of quartic couplings, i.e. 0.01, 0.1, 0.4 and 0.8, which are depicted by red, blue, orange and green curves, respectively. Similarly, for the plots in the second row, ${\widetilde{\lambda }}_2$ describes the variation in any particular generation, and the remaining generations are denoted by ${\widetilde{\lambda }}^{jj}$. The other quartic coupling terms $\lambda _2^{ii}$ are defined for all three generations of leptoquark. The variations are considered for lower and higher values of $Y_\phi$, i.e. 0.1 (left) and 1.0 (right)

In this section, we study the RG evolution of Higgs-lepton quark quartic couplings of leptoquark $\widetilde{R}_2$, i.e. $\lambda _{2}$ and ${\widetilde{\lambda }}_{2}$. As already mentioned, these terms should always remain below $4\pi$ to maintain the perturbativity of the theory. The one-loop beta functions for these two parameters are given below:

(58)

(59)

For the three-generation case, $\lambda _{2}$ and ${\widetilde{\lambda }}_2$ become two $3\times 3$ matrices whose ij-th element indicates the quartic coupling of the i-th and j-th generations of $\widetilde{R}_2$ with two Higgs fields. However, as mentioned earlier, we restrict our parameter space with no mixing among the generations of leptoquarks at the initial scale; therefore, $\lambda _{2}$ and ${\widetilde{\lambda }}_2$ become two diagonal matrices. The one-loop beta functions for these two parameters are simply given by:

(60)

The full two-loop beta functions for these two parameters with both one and three generations are presented in Appendix E.

Now, we study the variation in quartic coupling between the leptoquark and Higgs with perturbative scale, i.e. the scale at which any of the coupling diverges. The variations in the quartic couplings $\lambda _2$ and ${\widetilde{\lambda }}_2$ for three generations of doublet leptoquark are explained in Fig.4. In the first two plots, Fig. 4a, b, $\lambda _2$ corresponds to the quartic coupling term for one particular generation of leptoquark while $\lambda _2^{jj}$ denotes the remaining generations of $\lambda _2$, and all the generations of the other quartic coupling term ${\widetilde{\lambda }}_2$ are designated as ${\widetilde{\lambda }}_2^{ii}$. Similarly, for ${\widetilde{\lambda }}_2$ variation in Fig. 4c, d, ${\widetilde{\lambda }}_2$ corresponds to any particular generation of leptoquark while the remaining generations are denoted by ${\widetilde{\lambda }}_2^{jj}$, and the other quartic coupling terms $\lambda _2^{ii}$ signify $\lambda _{2}$ for all three generations. The plots in the left panel indicate a relatively low value of Yukawa, i.e. $Y_\phi =0.1$, whereas the same in the right panel illustrate the variation in the mentioned couplings for a higher value of Yukawa, i.e. $Y_\phi =1.0$.

In the first two plots, the initial value of $\lambda _2$ is varied from 0.1 to 0.8 keeping the values for other quartic couplings at EW scale as 0.01, 0.1, 0.4 and 0.8, which are depicted by red, blue, orange and green curves, respectively. As can be observed from Eqs. (58) and (60), the one-loop beta function of $\lambda _{2}$ receives enhanced contributions from positive-valued ${\widetilde{\lambda }}_{2}$, and hence $\lambda _{2}$ reaches non-perturbativity quickly for larger values of ${\widetilde{\lambda }}_{2}$. It should be noted from Fig: 4a that for $(\lambda _2^{jj},{\widetilde{\lambda }}_2^{ii})=0.01$ and 0.1 at the EW scale, the theory remains perturbative up to Planck scale for $\lambda _2\le 0.62$ and 0.52, respectively, with $Y_\phi =0.1$. As we increase the EW values to 0.4 and 0.8, the positive contribution from quartic couplings makes the theory non-perturbative at $\sim 10^{12}$ GeV, $10^{7}$ GeV for lower initial values of $\lambda _2$. For higher EW values of $\lambda _{2}$, this perturbative scale decreases slowly. The variation in $\lambda _2$ with perturbative scale for $Y_\phi =1.0$, as displayed in Fig. 4b, looks quite similar to the previous case. However, as can be seen from Eqs. (58) and (60), the one-loop beta function of $\lambda _{2}$ obtains positive contributions from the term (since and are negligible), and therefore $\lambda _{2}$ becomes non-perturbative at a slightly lower energy scale than in the previous case. In this case, $\lambda _2$ is bounded above to 0.56 and 0.47 for EW values of other quartic couplings $\lambda _2^{jj},{\widetilde{\lambda }}_2^{ii}$ to be 0.01 and 0.1, respectively. Further increases in EW values to 0.4 and 0.8 make the theory non-perturbative around $10^{11.2}$ GeV and $10^{6.9}$ GeV, respectively, for lower initial values of $\lambda _{2}$, and the scale diminishes gently with higher initial values of $\lambda _{2}$. It is worth mentioning that the non-perturbativity of $\lambda _{2}$ and ${\widetilde{\lambda }}_2$, attained with three generations of $\widetilde{R}_2$, is not a result of any Landau pole, which is also apparent from the different positioning of the non-perturbative scales compared to that of the gauge couplings.

In a similar fashion, ${\widetilde{\lambda }}_{2}$ is altered gradually from 0.1 to 0.8 in the last two plots, fixing the values of other quartic couplings as 0.01, 0.1, 0.4 and 0.8, which are depicted by red, blue, orange and green curves, respectively. In this case, $\lambda _{2}$ provides a positive effect in the running of ${\widetilde{\lambda }}_2$, see Eqs. (59) and (60); therefore, ${\widetilde{\lambda }}_2$ moves to the non-perturbative region more rapidly for higher values of $\lambda _{2}$. On the other hand, $Y_2$ also contributes positively through the term , and hence ${\widetilde{\lambda }}_2$ hits non-perturbativity at slightly lower energy scale for higher Yukawa coupling. Form Fig. 4(c) we see that the demand of Planck scale perturbativity constrains ${\widetilde{\lambda }}_2$ to be smaller than 0.8 and 0.66 if the EW values of other quartic couplings $({\widetilde{\lambda }}_2^{jj}, \lambda _2^{ii})$ are set as 0.01 and 0.1 respectively, with $Y_\phi =0.1$. With higher values of ${\widetilde{\lambda }}_2^{jj}, \lambda _2^{ii}$ at EW scale, i.e. 0.4 and 0.8, the model becomes non-perturbative at much lower energy than Planck scale. From Fig. 4d, in comparison with Fig. 4c, we observe that ${\widetilde{\lambda }}_2$ is restricted to slightly lower values, i.e. 0.73 and 0.59, if we begin with 0.01 and 0.1, respectively, for the EW values of other quartic couplings along with $Y_\phi =1.0$. The statement with higher initial values of the quartic couplings remains valid in this scenario as well.

3.2.2 Perturbativity of $\varvec{S}_3$

In this section, we scrutinize the RG evolution of Higgs-leptoquark quartic couplings for $\varvec{S}_3$, namely $\lambda _{3}$ and ${\widetilde{\lambda }}_{3}$. These two parameters also should be bounded above by $4\pi$. The one-loop beta functions for these two parameters in the one-generation case are given below:

(61)

(62)

Like the doublet leptoquark case, for the three-generation scenario, $\lambda _{3}$ and ${\widetilde{\lambda }}_{3}$ become two $3\times 3$ matrices whose ij-th element indicates the quartic coupling of i-th and j-th generations of $\varvec{S}_3$ with two Higgs fields. Nevertheless, as mentioned earlier, we have restricted our parameter space with no mixing among the generations of leptoquarks at the initial scale, making $\lambda _{3}$ and ${\widetilde{\lambda }}_3$ two diagonal matrices. The one-loop beta functions for these two parameters are given simply by:

(63)

The full two-loop beta functions for $\varvec{S}_3$ with both one and three generations are presented in Appendix E.

Fig. 5

Variation in leptoquark-Higgs quartic coupling $\lambda _3$ and ${\widetilde{\lambda }}_3$ for triplet leptoquark $\varvec{S}_3$ with the perturbative scale. For the plots in the first row, $\lambda _3$ variation is considered for any particular generation of leptoquark, and the same for the remaining generations are denoted by $\lambda _3^{jj}$. The other leptoquark-Higgs quartic couplings ${\widetilde{\lambda }}_3^{ii}$ include all three leptoquark generations. Similarly, for the plots in the second row, the variation in quartic coupling ${\widetilde{\lambda }}_3$ for any particular leptoquark generation is depicted while symbolizing the same for the remaining generations by ${\widetilde{\lambda }}_3^{jj}$. The other quartic coupling $\lambda _3^{ii}$ includes all three generations of leptoquark. The variations are considered for four different initial values, i.e. 0.01, 0.1, 0.4 and 0.8, at EW scale, which are described by red, blue, orange and green curves, respectively. Here, two different values for $Y_\phi$ are considered, which are 0.1 and 0.8

Now, we consider the variation in quartic coupling between the leptoquark $\varvec{S}_3$ and Higgs with perturbative scale, and it is illustrated in Fig. 5 for the case of three generations. In the first two plots, Fig. 5a, b, $\lambda _3$ corresponds to the quartic coupling term for one particular generation of leptoquark, while $\lambda _3^{jj}$ denote the remaining generations of $\lambda _3$, and all the generations of other quartic coupling terms ${\widetilde{\lambda }}_3$ are designated as ${\widetilde{\lambda }}_3^{ii}$. Similarly, for ${\widetilde{\lambda }}_3$ variation in Fig. 5c, d, ${\widetilde{\lambda }}_3$ corresponds to any particular generation of leptoquark, while the remaining generations are denoted by ${\widetilde{\lambda }}_3^{jj}$, and the other quartic coupling terms $\lambda _3^{ii}$ signify $\lambda _{3}$ for all three generations. The plots in the left panel indicate a relatively low value of Yukawa, i.e. $Y_\phi =0.1$, whereas the same in the right panel illustrate the variation in the mentioned couplings for a higher value of Yukawa, i.e. $Y_\phi =0.8$.

Fig. 6

Variation in quartic coupling $\lambda _2, {\widetilde{\lambda }}_2, \lambda _3$ and ${\widetilde{\lambda }}_3$ with perturbative scale for three generations of ${\widetilde{R}}_2 + \varvec{S}_3$. Here, the variation in $\lambda _2$ is shown for any one generation of ${\widetilde{R}}_2$, and the remaining generations for the $\lambda _2$ term of $\varvec{R}_2$ are defined by $\lambda _\beta ^{jj}$. The $\lambda _\alpha ^{ii}$ term corresponds to three generations of ${\widetilde{R}}_2$ for the ${\widetilde{\lambda }}_2$ term and three generations of $\varvec{S}_3$ for the $\lambda _3$ and ${\widetilde{\lambda }}_3$ term. Again, the ${\widetilde{\lambda }}_2$ variation is depicted for any one generation of ${\widetilde{R}}_2$, and the remaining generations of ${\widetilde{R}}_2$ are given by $\lambda _\beta ^{jj}$ for the ${\widetilde{\lambda }}_2$ terms. In this case, the $\lambda _\alpha ^{ii}$ term corresponds to three generations of ${\widetilde{R}}_2$ for the $\lambda _2$ terms and three generations of $\varvec{S}_3$ for the $\lambda _3$ and ${\widetilde{\lambda }}_3$ terms. Similar notation is followed for the variation in $\lambda _3$ and ${\widetilde{\lambda }}_3$. The EW scale values for the quartic couplings other than the coupling whose variation is considered are set to four different values, i.e. 0.01, 0.1, 0.4 and 0.8, which are illustrated by red, blue, orange and green curves, respectively, taking $Y_\phi$=0.1. Here, $Y_\phi$ signifies both $Y_2$ and $Y_3$ with three generations

In the first two plots, Fig. 5a, b, we have gradually varied the initial values for $\lambda _{3}$ from 0.1 to 0.8, keeping the EW values for other quartic couplings as 0.01, 0.1, 0.4 and 0.8, respectively, which are presented by red, blue, orange and green lines. Similar conditions for ${\widetilde{\lambda }}_{3}$ are presented in Fig. 5c, d. As we already showed in the earlier sections, all the other couplings for three generations of the triplet leptoquark diverge at $10^{19.7}$ GeV due to the gauge coupling $g_2$. The couplings $\lambda _{3}$ and ${\widetilde{\lambda }}_{3}$ are also not different from that behaviour. Therefore, unlike the $\widetilde{R}_2$ case, here $\lambda _{3}$ and ${\widetilde{\lambda }}_3$ diverge at $10^{19.7}$ GeV for any smaller initial values of $\lambda _{3}$ and ${\widetilde{\lambda }}_{3}$ at EW scale with any value of Yukawa coupling $Y_\phi$. Now, as can be observed from Eqs. (61) and (63), ${\widetilde{\lambda }}_{3}$ contributes positively in the one-loop beta function of $\lambda _{3}$, and hence $\lambda _{3}$ reaches non-perturbativity at an early stage with higher values of ${\widetilde{\lambda }}_{3}$. On the other hand, due to the positive effect of at one-loop order, all the lines shift slightly downward with higher values of Yukawa couplings, but the shifts are almost unnoticeable. Both of the above statements are also true for the running of ${\widetilde{\lambda }}_{3}$. For $\lambda _{3}$, Planck scale perturbativity is achieved until 0.51 and 0.37, with other quartic coupling at EW scale being 0.01 and 0.1, respectively, for both the Yukawa coupling. However, for higher values of other quartic couplings at EW scale, i.e. 0.4 and 0.8, $\lambda _3$ diverges at much lower scale, i.e. $10^{10.8}$ GeV and $10^{6.4}$ GeV, with its lower initial values, and this decreases with the increase in the beginning value of $\lambda _3$. Likewise, the quartic coupling ${{\widetilde{\lambda }}}_3$ is constrained to 0.76 and 0.52 for Planck scale perturbativity, with EW values of other quartic couplings at 0.01 and 0.1, respectively. For higher EW values of quartic couplings, the theory becomes non-perturbative at much lower scales as previously discussed.

3.2.3 Perturbativity of $\widetilde{R}_2+\varvec{S}_3$ with 3-gen

Now, we move to the combined scenario of $\widetilde{R}_2$ and $\varvec{S}_3$ with three generations. The one-loop beta functions for all the Higgs-leptoquark quartic couplings in this case can easily be written as:

$$\begin{aligned}&\beta (\lambda _{2}^{ii})^{1\mathrm{-loop}}_{\widetilde{R}_2+\varvec{S}_3,3\mathrm{-gen}}=\beta (\lambda _{2}^{ii})^{1\mathrm{-loop}}_{\widetilde{R}_2,3\mathrm{-gen}},\nonumber \\&\beta ({\widetilde{\lambda }}_{2}^{ii})^{1\mathrm{-loop}}_{\widetilde{R}_2+\varvec{S}_3,3\mathrm{-gen}}=\beta ({\widetilde{\lambda }}_{2}^{ii})^{1\mathrm{-loop}}_{\widetilde{R}_2,3\mathrm{-gen}},\nonumber \\&\beta (\lambda _{3}^{ii})^{1\mathrm{-loop}}_{\widetilde{R}_2+\varvec{S}_3,3\mathrm{-gen}}=\beta (\lambda _{3}^{ii})^{1\mathrm{-loop}}_{\varvec{S}_3,3\mathrm{-gen}},\nonumber \\&\beta ({\widetilde{\lambda }}_{3}^{ii})^{1\mathrm{-loop}}_{\widetilde{R}_2+\varvec{S}_3,3\mathrm{-gen}}=\beta ({\widetilde{\lambda }}_{3}^{ii})^{1\mathrm{-loop}}_{\varvec{S}_3,3\mathrm{-gen}}. \end{aligned}$$

(64)

The full two-loop beta functions of all the Higgs-leptoquark quartic couplings in this scenario are listed in Appendix G.

Fig. 7

Variation in quartic couplings $\lambda _2$ and ${\widetilde{\lambda }}_2$ with the perturbative scale for one generation of ${\widetilde{R}}_2 + \varvec{S}_3$. Here, for the $\lambda _2$ variation, $\lambda _\alpha$ corresponds to {${\widetilde{\lambda }}_2, \lambda _3$, ${\widetilde{\lambda }}_3$}, and if we consider the behaviour of ${\widetilde{\lambda }}_2$, then $\lambda _\alpha$ includes $\lambda _2, \lambda _3$ and ${\widetilde{\lambda }}_3$. All quartic couplings other than that for which the variation is considered are assigned four different values at the EW scale, i.e. 0.01, 0.1, 0.4 and 0.8, and these are delineated by red, blue, orange and green curves, respectively. Two different values of $Y_\phi$ are i.e. 0.1 and 0.8, with $Y_\phi$ representing the Yukawa couplings for both the leptoquarks

Fig. 8

Variation in triplet leptoquark-Higgs quartic coupling $\lambda _3$ and ${\widetilde{\lambda }}_3$ for one generation of ${\widetilde{R}}_2 + \varvec{S}_3$. Here, for the $\lambda _3$ variation, $\lambda _\alpha \in \{\lambda _2, {\widetilde{\lambda }}_2, {\widetilde{\lambda }}_3\}$, and for ${\widetilde{\lambda }}_3$, $\lambda _\alpha \in \{\lambda _2, {\widetilde{\lambda }}_2, \lambda _3\}$. We consider four different values of $\lambda _\alpha$ at the EW scale, i.e. 0.01, 0.1, 0.4 and 0.8, which are denoted by red, blue, orange and green curves, respectively. The black dotted line parallel to the x-axis denotes the Planck scale

For three generations of $\varvec{S}_3 + {\widetilde{R}}_2$, we have already seen that all the gauge couplings diverge below Planck scale, i.e. at $10^{14.4}$ GeV, mainly due to the typical behaviour of $g_2$ at two-loop order. This affects the running of quartic couplings as well. We study the variation in these couplings with perturbative scale in Fig. 6, assuming $Y_\phi =0.1$. The adjustments in these plots with larger $Y_\phi$ are not very significant, and hence we do not present them. When examining the variation in $\lambda _{2}$ for any particular generation, the remaining generations of $\lambda _{2}$ are denoted as $\lambda _\beta ^{jj}$, whereas the other quartic couplings including ${\widetilde{\lambda }}_2$, $\lambda _{3}$ and ${\widetilde{\lambda }}_{3}$ with all the generations are designated as $\lambda _\alpha ^{ii}$. The same notation is followed for all the other quartic couplings. The colour codes are as discussed previously. It can be observed from Fig. 6a–c that even for lower initial values of $\lambda _{\alpha }^{ii}$ and $\lambda _{\beta }^{jj}$, i.e. 0.01 and 0.1, the quartic couplings enter the non-perturbative region at $10^{14.4}$ GeV due to the appearance of the Landau pole in $g_2$. For higher values of the parameters at EW scale, non-perturbativity is reached even at much lower scale. Thus, the demand of Planck scale perturbativity rules out the three-generation scenario of the $\varvec{S}_3 + {\widetilde{R}}_2$ model for any values of the leptoquark-Higgs quartic couplings. Therefore, we have to consider the one-generation scenario of the $\varvec{S}_3 + {\widetilde{R}}_2$ model.

3.2.4 Perturbativity of $\widetilde{R}_2+\varvec{S}_3$ with 1-gen

In this section, we look into the perturbativity of Higgs-leptoquark quartic couplings for the combined scenario of $\widetilde{R}_2$ and $\varvec{S}_3$ with one generation. The one-loop beta functions for all these parameters in this case can easily be written as:

$$\begin{aligned}&\beta (\lambda _{2})^{1\mathrm{-loop}}_{\widetilde{R}_2+\varvec{S}_3,1\mathrm{-gen}}=\beta (\lambda _{2})^{1\mathrm{-loop}}_{\widetilde{R}_2,1\mathrm{-gen}}, \nonumber \\&\beta ({\widetilde{\lambda }}_{2})^{1\mathrm{-loop}}_{\widetilde{R}_2+\varvec{S}_3,1\mathrm{-gen}}=\beta ({\widetilde{\lambda }}_{2})^{1\mathrm{-loop}}_{\widetilde{R}_2,1\mathrm{-gen}},\nonumber \\&\beta (\lambda _{3})^{1\mathrm{-loop}}_{\widetilde{R}_2+\varvec{S}_3,1\mathrm{-gen}}=\beta (\lambda _{3})^{1\mathrm{-loop}}_{\varvec{S}_3,1\mathrm{-gen}},\nonumber \\&\beta ({\widetilde{\lambda }}_{3})^{1\mathrm{-loop}}_{\widetilde{R}_2+\varvec{S}_3,1\mathrm{-gen}}=\beta ({\widetilde{\lambda }}_{3})^{1\mathrm{-loop}}_{\varvec{S}_3,1\mathrm{-gen}}. \end{aligned}$$

(65)

The full two-loop beta functions of all the Higgs-leptoquark quartic couplings in this scenario are listed in Appendix G.

Now, we study the variation in leptoquark-Higgs quartic couplings $\lambda _2, {\widetilde{\lambda }}_2$, $\lambda _3$ and ${\widetilde{\lambda }}_3$ with the perturbative scale for the one generation of ${\widetilde{R}}_2 + \varvec{S}_3$ model. The results for the variation in $\lambda _{2}$ and ${\widetilde{\lambda }}_{2}$ are presented in Fig. 7. When considering the variation in $\lambda _2$, we denote all the other leptoquark-Higgs quartic couplings, namely ${\widetilde{\lambda }}_2, \lambda _3$ and ${\widetilde{\lambda }}_3$, as $\lambda _\alpha$. By the same token, when examining the behaviour of ${\widetilde{\lambda }}_2$, the other leptoquark-Higgs quartic couplings, i.e. $\lambda _2, \lambda _3$ and ${\widetilde{\lambda }}_3$, are taken as $\lambda _\alpha$. The colour codes are as described in earlier sections. As can be seen from Fig. 7a, b, the initial value of $\lambda _2$ is restricted to 0.62 and 0.54 from Planck scale perturbativity for EW values of other quartic couplings of 0.01 and 0.1, respectively, with $Y_\phi =0.1$, whereas with $Y_\phi =0.8$, these upper bounds roll down to 0.59 and 0.51, respectively. For higher values of other quartic couplings at the EW scale, i.e. 0.4 and 0.8, the theory becomes non-perturbative around $10^{14.1}$ GeV and $10^{7.9}$ GeV with $Y_\phi =0.1$, which differ slightly (about 0.2 GeV) in the $Y_\phi =0.8$ case even if the initial value of $\lambda _{2}$ is taken to be very small. Similarly, for ${\widetilde{\lambda }}_2$, Planck scale perturbativity with $Y_\phi =0.1$ is achieved up to ${\widetilde{\lambda }}_2 \le$ 0.82 and 0.68, which decrease to 0.76 and 0.63, respectively, with $Y_\phi =0.8$, while taking the initial values for other quartic couplings as 0.01 and 0.1 at EW scale. Again, for higher EW values of $\lambda _\alpha$, i.e. 0.4 and 0.8, the theory becomes non-perturbative at much lower scales, as shown in Fig. 7c, d. The reason for all these typical behaviours were already discussed in the previous Sect. 3.2.1.

Correspondingly, the changes in $\lambda _{3}$ and ${\widetilde{\lambda }}_{3}$ with perturbative scale are displayed in Fig. 8. Here, for $\lambda _3$ variation, we designate $\{\lambda _2, {\widetilde{\lambda }}_2, {\widetilde{\lambda }}_3\}$ as $\lambda _\alpha$, whereas for ${\widetilde{\lambda }}_3$ variation, we assume $\lambda _\alpha \in \{\lambda _2, {\widetilde{\lambda }}_2, \lambda _3\}$. The colour codes have already been discussed. Here, Plank scale perturbativity with $Y_\phi =0.1$ restricts $\lambda _3$ to 0.55 and 0.47 (see Fig. 8a), which change to 0.53 and 0.45, respectively, with $Y_\phi =0.8$ (Fig. 8b), for $\lambda _\alpha$=0.01, 0.1 at the EW scale. Similarly, from Fig. 8c, d, one can observe that for $Y_\phi =0.1$, ${\widetilde{\lambda }}_{3}$ should be bounded above to 0.83 and 0.67, which reduce to 0.78 and 0.62, respectively, for $Y_\phi =0.8$, in order to respect Plank scale perturbativity with $\lambda _\alpha$=0.01, 0.1 at the EW scale. For higher initial values of $\lambda _\alpha$, i.e. 0.4 and 0.8, the theory becomes non-perturbative at very low scale, i.e. $\sim$ $10^{14-15}$ GeV and $10^{8-9}$ GeV, even with very small EW value of $\lambda _{3}$ and ${\widetilde{\lambda }}_3$ at both the Yukawa couplings, and the scale decreases gradually with the increase in initial values of these two parameters. The reason for all these typical behaviours were already discussed in the previous Sect. 3.2.2. It is worth noting again that there is no Landau pole of any gauge coupling in this model, and the non-perturbativity discussed here appears because of the Higgs-leptoquark quartic couplings growing beyond $4\pi$ during the RG evolution.

3.2.5 Effects of self-quartic couplings of leptoquarks

Up to this point, for the sake of simplicity, we have not considered self-quartic couplings of the leptoquark. In this subsection, we discuss the effects of such couplings on the perturbativity of the model. We find that the introduction of these couplings has little affect on the running of gauge couplings; however, it brings in a non-negligible positive contribution to the running of Higgs-leptoquark quartic couplings up to two-loop order. Therefore, Higgs-leptoquark quartic couplings attain the non-perturbative limit earlier than in the scenario wherein self-quartic couplings of leptoquarks are neglected. For instance, one can add the self-interaction term of $\omega _2 (\widetilde{R}_2^\dagger \widetilde{R}_2)(\widetilde{R}_2^\dagger \widetilde{R}_2)$ to the Lagrangian given by Eq. 1. With values of 0.47 and 0.64 at EW scale, $\lambda _2$ enters the non-perturbative region at Planck scale in this case for other quartic couplings $\lambda _{2}^{jj}$, ${\widetilde{\lambda }}_2^{ii}$ and the newly introduced self-quartic coupling of ${\widetilde{R}}_2$ (three generations, without any generation mixing) being 0.01 and 0.1, respectively, assuming $Y_\phi =1.0$. Before the introduction of self-quartic coupling of ${\widetilde{R}}_2$, the values of $\lambda _{2}$ for which non-perturbativity was achieved at Planck scale under the same values of other quartic couplings were 0.47 and 0.31, respectively (see Fig. 4b). With the same value of $Y_\phi$ and other quartic couplings, ${{\widetilde{\lambda }}}_{2}$ maintains Planck scale perturbativity until values at EW scale of 0.64 and 0.41, which were 0.73 and 0.58, respectively (see Fig. 4d), before the introduction of self-quartic coupling of ${\widetilde{R}}_2$ (three generations, without any generation mixing). On the other hand, for $\varvec{S}_3$ with three generations, the positive effects of self-quartic couplings of leptoquarks are even stronger. As an example, we add a self-quartic term⁶ of $\text {Tr}\big [(S_3^{ad})^\dagger S_3^{ad}\big ]^2$ to the Lagrangian given by Eq. 4. With $Y_\phi =0.8$, the parameters $\lambda _{3}$ and ${{\widetilde{\lambda }}}_{3}$ now cannot achieve Planck scale perturbativity for small values of other quartic couplings such as 0.01 at EW scale. Before the consideration of self-quartic coupling of the leptoquark, $\lambda _{3}$ and ${{\widetilde{\lambda }}}_{3}$ achieved Planck scale perturbativity with other quartic couplings of 0.1 at EW scale (see Fig. 5b, d).

4 Vacuum stability

There exist two approaches in the literature regarding stability analysis. The first is the running of Higgs quartic coupling $\lambda _h$ using beta functions, and the other method is the Coleman–Weinberg effective potential approach [145].

First, we scrutinize the running of self-quartic coupling for the Higgs boson, i.e. $\lambda _{h}$, which in turn would indicate the change in stability of the Higgs vacuum. This parameter is also expected to be below $4\pi$ in the entire energy scale to respect the perturbativity. However, for the purpose of this section, we focus on the stability of the vacuum, which suggests that $\lambda _{h}$ should be a positive quantity throughout the energy scale. The one- and two-loop beta functions for $\lambda _{h}$ under the SM are given by:

(66)

(67)

It is well known that in the case of the SM, $\lambda _h$ enters the negative-valued region between the $10^{9}$ GeV and $10^{10}$ GeV energy scale [144, 146] at two-loop order. At this point it is worth mentioning that in the case of $\lambda _{h}$, the two-loop contributions affect the running significantly. The addition of right-handed neutrinos pulls the stability scale further down with more negative contributions [115,116,117,118,119]. In contrast, the presence of scalar leptoquarks is expected to further push the stability scale by adding positive contributions to these beta functions.

4.1 Vacuum stability of ${\widetilde{R}}_2$

Fig. 9

Running of SM Higgs quartic coupling with scale for stability analysis. Stability scale is defined as the scale after which $\lambda _h<0$. Here, blue, yellow and red curves describes the running of $\lambda _h$ for SM, one generation and three generations of ${\widetilde{R}}_2$. With three generations of ${\widetilde{R}}_2$, the positive contribution from gauge couplings and other quartic couplings is large enough and is compensated by the negative contribution of $Y_2$. The crucial value of $Y_2$ is 1.36; $\lambda _h$ will go negative if higher values of $Y_2$ are considered and stability is lost. On the other hand, for one generation of $\widetilde{R}_2$, $\lambda _{h}$ always goes to a negative value before Planck scale

First, we look into the effects of the doublet leptoquark ${\widetilde{R}}_2$. The one- and two-loop beta functions for $\lambda _{h}$ in this case are given by:

(68)

(69)

In the last section, we observed that there is not much room for the Higgs-leptoquark quartic couplings to be varied randomly from the perspective of Planck scale perturbativity. However, the Yukawa couplings for leptoquarks do not attain such serious constraints. Therefore, we address the issue of vacuum stability from the effects of leptoquark Yukawa coupling. However, it should be noted from Eqs. (68) and (69) that the contributions of $Y_2$ appear at the two-loop level only. The effects of $Y_2$ in the running of $\lambda _h$ for ${\widetilde{R}}_2$ with both one-generation and three-generation cases are shown in Fig. 9. Here, the blue, yellow and red curves explain the running of $\lambda _h$ for SM, one generation of ${\widetilde{R}}_2$ and three generations of ${\widetilde{R}}_2$, respectively. For all the analyses, we assume every Higgs-leptoquark quartic coupling to be 0.01.

Fig. 10

Running of Higgs quartic coupling with scale for triplet leptoquark $\varvec{S}_3$ at two-loop level. Here, $\lambda _h$ running for SM, one generation and three generations of $\varvec{S}_3$ is delineated by blue, yellow and red curves, respectively. If $Y_\phi$ is assumed to be greater than 1.29, the one-generation model of $\varvec{S}_3$ loses stability at two-loop order, although the three-generation scenario remains stable. However, if we consider $Y_\phi >3.9$, the three-generation scenario of the $\varvec{S}_3$ model also leaves the stable region at two-loop order

As already mentioned, the stability scale after which $\lambda _{h}$ turns negative for the SM is just above $10^9$ GeV at the two-loop level. However, when considering the RG evolution of $\lambda _{h}$ in the $\widetilde{R}_2$ case, the gauge couplings and other quartic couplings contribute positively whereas the Yukawa coupling of the leptoquark inserts negative contributions at the two-loop level, as can be seen from Eqs. (68) and (69). Again, since the additional contribution in the beta function of $\lambda _{h}$ for the case of three generations is the sum of all individual generations, and the gauge couplings at any particular scale for the three-generation case are higher than the same for the one-generation case, the three-generation scenario obtains more positive contributions than the one-generation case. It should also be noted that although there are two negative and three positive terms containing in Eq. (69), the positive terms are quadratic in and , and therefore smaller than the negative terms which are linear in and . Thus the yellow curve representing leptoquark $R_2$ with one generation stays above the blue line depicting the SM, and the red curve signifying $R_2$ with three generations lies further above the region. However, due to the negative contributions of the Yukawa couplings of the leptoquark, the red and yellow line move downward with the enhancement in $Y_\phi$. In Fig. 9a, b, we depict the variations in $\lambda _{h}$ with the energy scale, taking the initial values for $Y_\phi$ as 1.0 and 1.36, respectively. As can be observed, for both cases the vacuum of the leptoquark model $\widetilde{R}_2$ with one generation remains stable up to $\sim 10^{9.5}$ GeV, slightly higher than the SM estimates. However, it is interesting to observe in the left panel that the red curve remains in the positive region of $\lambda _{h}$ for all energy values, indicating the stability of the vacuum all the way to Planck scale with three generations of ${\widetilde{R}}_2$ and $Y_\phi =1.0$. Once we start with an initial value of $Y_\phi$ of 1.36, we observe in the right panel that the red curve touches the $\lambda _{h}=0$ line, and thus for higher values of $Y_\phi$ the Planck scale stability will be lost. One can also observe that for this particular value of $Y_\phi$, the red curve touches the $\lambda _{h}=0$ line at $\sim 10^{14.5}$ GeV and remains very flat up to the Planck scale. One can also find that this value of $Y_\phi$ of 1.36 is higher than the required Yukawa coupling $Y_N$ in inert doublet+type III seesaw or inverse type III seesaw to maintain the Planck scale stability [120]. On the contrary, it should be noted that leptoquark $\widetilde{R}_2$ with one generation does not show Planck scale stability even with very low Yukawa. Now, it is worth mentioning that with the change in Higgs-leptoquark couplings from 0.01 to 0.1, we do not find any significant changes in the behaviour of $\lambda _{h}$. Although very high values of $\lambda _2$ and ${\widetilde{\lambda }}_2$ might shift the red curve in an upward direction, these higher values are disfavoured from Planck scale perturbativity of $\lambda _2$ and ${\widetilde{\lambda }}_2$. Consideration of the self-quartic coupling of the leptoquark introduces positive contributions, indicating the need for a higher initial value of $Y_\phi$ to push $\lambda _{h}$ to the negative region. However, for $\widetilde{R}_2$ with three generations, we find little difference in the critical value of $Y_\phi$.

4.2 Vacuum stability of $\varvec{S}_3$

Now, we discuss the stability of Higgs vacuum for the $\varvec{S}_3$ scenario. The one- and two-loop beta functions of $\lambda _h$ in this case are as follows:

(70)

(71)

The running of Higgs quartic coupling for the triplet leptoquark $\varvec{S}_3$ is displayed in Fig. 10 taking the EW value of $\lambda _3$ and ${\widetilde{\lambda }}_{3}$ as 0.01. Here, blue, yellow and red curves denote the RG evolution of $\lambda _h$ for SM, one generation of $\varvec{S}_3$ and three generations of $\varvec{S}_3$, respectively, at two-loop order. As discussed in the previous section 4.1, gauge couplings and Higgs-leptoquark quartic couplings contribute positively in the running of $\lambda _{h}$, while the leptoquark Yukawa coupling brings in negative effects (see Eqs. (70) and (71)). Furthermore, since $\widetilde{R}_2$ lies in fundamental representation of $SU(2)_L$, while $\varvec{S}_3$ stays in adjoint representation, the positive effects in the case of $\varvec{S}_3$ are very large compared to the same for $\widetilde{R}_2$; therefore, large Yukawa coupling for $\varvec{S}_3$ will be needed to make the vacuum unstable. We show the results for $Y_\phi$ of 1.29 and 3.9 in Fig. 10a, b, respectively. In the left panel, we see that both the case of $\varvec{S}_3$ with one generation and that with three generations show Planck scale stability for $Y_\phi =1.29$, but the yellow curve touches the $\lambda _{h}=0$ line, implying that a further increase in $Y_\phi$ will make the theory unstable before the Planck is reached. It is also interesting to note that the yellow curve touches the $\lambda _{h}=0$ line at $\sim 10^{15}$ GeV and remains very flat up to the Planck scale as in the $\widetilde{R}_2$ scenario with three generations. In the right panel, one can observe that the higher value of $Y_\phi$, i.e. 3.9, has forced the red and yellow curves to move downward pushing the one-generation $\varvec{S}_3$ to the unstable region. However, the red curve touches the $\lambda _{h}=0$ line at this Yukawa coupling, indicating that $Y_\phi \le 3.9$ in order to preserve Planck scale stability with three generations of $\varvec{S}_3$. It is worth mentioning that here the red curve just kisses the $\lambda _{h}=0$ line at a lower energy scale of $\sim 10^{13.5}$ GeV, and then the positive contributions cause it to grow faster in the positive direction, unlike the previous cases. However, to ensure perturbativity of the model, $Y_\phi \le \sqrt{4\pi }\approx 3.54$. Therefore, combining vacuum stability and perturbativity, one should consider $\sqrt{4\pi }$ as the upper limit of $Y_\phi$ for the three-generation scenario of $\varvec{S}_3$. Like $\widetilde{R}_2$, in this case also, the behaviour of these plots does not show any notable alteration if $\lambda _3$ and ${\widetilde{\lambda }}_{3}$ are increased to 0.1 from 0.01. The inclusion of leptoquark self-quartic coupling inserts huge positive effects for $\varvec{S}_3$. Therefore, with three generations of $\varvec{S}_3$, the critical initial value of 3.9 for $Y_\phi$ now goes beyond 5. However, since $\sqrt{4\pi }\le 5$, the upper bound on $Y_\phi$ remains $\sqrt{4\pi }$ when considering combined constraint from vacuum stability and perturbativity.

4.3 Vacuum stability of $\widetilde{R}_2+\varvec{S}_3$ with 3-gen

Fig. 11

Running of Higgs quartic coupling $\lambda _h$ for three generations of ${\widetilde{R}}_2 + \varvec{S}_3$ at the two-loop level. Here, all the leptoquark Yukawa couplings are assumed to be 1.0 and all the Higgs-leptoquark couplings are taken to be 0.01. In this model, $\lambda _h$ diverges at an energy scale ($\sim$ $10^{14.4}$ GeV) far below the Planck scale at two-loop order

The one-loop and two-loop beta functions for $\lambda _{h}$ with three generations of $\widetilde{R}_2+\varvec{S}_3$ can be written as:

(72)

The result at two-loop order for this scenario with all the leptoquark Yukawa couplings being 1.0 and all the Higgs-leptoquark couplings being 0.01 are shown in Fig. 11. We have already seen that in this model all the parameters blow up at the energy scale of $10^{14.4}$ GeV. The parameter $\lambda _{h}$ is also no different from them. With any value of Higgs-leptoquark coupling or Yukawa coupling less than one, this divergence is unavoidable for this model. It is also noteworthy that $\lambda _{h}$ grows into the non-perturbative region before the emergence of instability in this model. Therefore, we will discuss the behaviour of $\lambda _h$ for one generation of ${\widetilde{R}}_2 + \varvec{S}_3$.

4.4 Vacuum stability of $\widetilde{R}_2+\varvec{S}_3$ with 1-gen

The one- and two-loop beta functions for $\lambda _{h}$ for the combined scenario of $\widetilde{R}_2+\varvec{S}_3$ with one generation can simply be expressed as:

$$\begin{aligned}&\beta (\lambda _{h})^{1\mathrm{-loop}}_{\widetilde{R}_2+\varvec{S}_3,1\mathrm{-gen}}= \beta (\lambda _{h})^{1\mathrm{-loop}}_{\mathrm{SM}}+\varDelta \beta (\lambda _{h})^{1\mathrm{-loop}}_{\widetilde{R}_2}\nonumber \\&\quad +\varDelta \beta (\lambda _{h})^{1\mathrm{-loop}}_{\varvec{S}_3},\nonumber \\&\beta (\lambda _{h})^{2\mathrm{-loop}}_{\widetilde{R}_2+\varvec{S}_3,1\mathrm{-gen}}= \beta (\lambda _{h})^{2\mathrm{-loop}}_{\mathrm{SM}}+\varDelta \beta (\lambda _{h})^{2\mathrm{-loop}}_{\widetilde{R}_2}\nonumber \\&\quad +\varDelta \beta (\lambda _{h})^{2\mathrm{-loop}}_{\varvec{S}_3}. \end{aligned}$$

(73)

The two-loop result for this case with all the leptoquark-Higgs coupling being 0.01 is displayed in Fig: 12, where the blue curve represents the SM and the yellow line denotes this particular model. As can be seen, with $Y_\phi =1.0$, $\lambda _{h}$ stays entirely in the positive region, whereas for $Y_\phi \ge 1.21$, this model no longer remains stable. With $Y_\phi =1.21$, the orange curve touches the $\lambda _{h}=0$ line at a relatively higher scale, $\sim 10^{16}$ GeV, and remains mostly flat up to Planck scale. Here, $Y_\phi$ includes the leptoquark Yukawa couplings for both ${\widetilde{R}}_2$ and $\varvec{S}_3$. The result remains almost the same with all the leptoquark-Higgs coupling being 0.1.

Fig. 12

Higgs quartic coupling running with scale is given for one generation of ${\widetilde{R}}_2 + \varvec{S}_3$ . Here, $\lambda _h$ running for SM and one 0 of ${\widetilde{R}}_2 + \varvec{S}_3$ are explained by blue and yellow curves, respectively. In order to maintain Planck scale stability, the upper bound on $Y_\phi$ for this model is 1.21. Here, $Y_\phi$ includes the leptoquark Yukawa couplings for both ${\widetilde{R}}_2$ and $\varvec{S}_3$

4.5 Bounds from effective potential stability constraints

Now, to study the stability, we follow the Coleman–Weinberg effective potential approach [145], where the one-loop contributions from all the particles at zero temperature with vanishing moments are included in effective coupling $\lambda _\mathrm{eff}$. The effective potential for high field values in the h-direction can be defined as

$$\begin{aligned} V_\mathrm{eff}(h,\mu ) \ \simeq \ \lambda _\mathrm{eff}(h,\mu )\frac{h^4}{4},\quad \mathrm{with}~h\gg v \, , \end{aligned}$$

(74)

The possibility of minima in the leptoquark direction can lead to charge- and colour-breaking minima, which is physically unwanted. However, such possibilities have little relevance in our case. Firstly, unlike the Higgs field, the bare mass term for the leptoquark is chosen sufficiently large and positive, ensuring a positive sign of the effective leptoquark mass term, i.e. for $\widetilde{R}_2$, $m^2_2 + \lambda _{2}\frac{v^2}{2}, m^2_2 + (\lambda _{2}+ {\tilde{\lambda }}_2\frac{v^2}{2} > 0$, which gives $<\widetilde{R}_2>=0$, for both with and without self-leptoquark couplings at the tree level. The chances of a non-zero vacuum expectation value (VEV) at the loop level in the presence of the self-quartic coupling and with the negative Higgs-leptoquark quartic coupling, though possible, but for the choice of large positive bare leptoquark mass term, which is of the order of TeV, are diminished in our case. It should be noted that the possibility of the resultant negative mass gives rise to the unphysical solution.

Such observations have also been made in the context of 2HDM, that if $v_1\gg v_2$, where $v_{1,2}$ are the two VEVs corresponding to the two Higgs doublets $\varPhi _{1,2}$, the potential along the $\varPhi _2$ direction remains almost flat, and hence it is instructive to show the variation in the potential perpendicular to it, i.e. along $\varPhi _1$ [147, 111, 118]. Even at the one-loop $\phi _2$ direction, it is not possible to have any deeper minima as compared to the $\phi _1$ direction. Similarly, in our leptoquark case, as the tree-level VEV in the leptoquark direction is zero, the possibility of deeper minima in that direction also ceases to exist.

The total potential including tree-level potential and the one-loop contributions from SM particles and leptoquarks can be defined as:

$$\begin{aligned} V= V_0 + V_1^{\mathrm{SM}} + V_1^{{\widetilde{R}}_2 / \varvec{S}_3 / {\widetilde{R}}_2 + \varvec{S}_3}, \end{aligned}$$

(75)

where $V_0$ is the tree-level potential of the model and $V_1$ is the one-loop effective potential which includes the contributions from the SM particles and the leptoquarks, and can be expressed as:

$$\begin{aligned} V_1(h, \mu ) \ = \ \frac{1}{64\pi ^2}\sum _{i} (-1)^F n_i {\mathcal {M}}_i^4(h) \Bigg [\log \frac{{\mathcal {M}}_i^2(h)}{\mu ^2}-c_i\Bigg ]. \end{aligned}$$

(76)

Here, the summation includes all the particles which couple to Higgs field h at tree level, where $n_i$ denotes the number of degrees of freedom for those particles, $c_i$ is a constant taking value $\frac{5}{6}$ for gauge bosons and $\frac{3}{2}$ for fermions and scalars, and the quantity F is another constant which becomes 0 for bosons and 1 for fermions. The entity ${\mathcal {M}}_i$, which is given by

$$\begin{aligned} {\mathcal {M}}_i^2(h) \ = \ \kappa _i h^2-\kappa '_i, \end{aligned}$$

(77)

signifies the field-dependent masses for the particles in the model, with $\kappa$ and $\kappa ^\prime$ being two constants. All the particles relevant for this paper are listed in Table 2 along with all the corresponding constants. For the numerical analysis, we have considered $h=\mu$, since the potential remains invariant at this scale [147].

Table 2 Different particles and the corresponding coefficients which contribute to the Coleman–Weinberg effective potential cf. Eq. (76). Here, the number of degrees of freedom for three generations of leptoquarks, i.e. 18, is shown outside the parentheses, while the same with one generation of leptoquark, i.e. 6, is listed inside the brackets

Fig. 13

Phase space diagram with Higgs mass $M_h$ vs top mass $M_t$ in GeV for $\widetilde{R}_2$. Green, yellow and red colours correspond to stable, metastable and unstable regions, respectively. The black dotted circles denote 1$\sigma$, 2$\sigma$ and 3$\sigma$ contours, and the black dot denotes the current Higgs mass and top mass value

Fig. 14

Phase diagram for $\varvec{S}_3$ with $M_h$ in GeV vs $M_t$ in GeV. The stable, metastable and unstable regions are delineated by green, yellow and red colours, respectively. The black dot denotes the current values of Higgs mass and top mass in GeV, and black circles are 1$\sigma$, 2$\sigma$ and 3$\sigma$ contours

The full effective potential in (75) can be redefined in terms of an effective quartic coupling $\lambda _\mathrm{eff}$, as in (74) using one-loop potential (76) as follows:

(78)

Now let us consider that there are two minima of the Higgs potential and we reside at the first one. If the second minimum is higher than the first one, the tunnelling from first minimum to the second one will be impossible, which in turn would indicate that the first minimum lies in the stable region, denoted by $\lambda _\mathrm{eff}>0$. But if the height of the second minimum is lower than that of the first one, there would be a finite probability for the system to tunnel to the second one. In this scenario, if the tunnelling lifetime becomes greater than the age of the universe, we term the first minimum as metastable region.

The tunnelling probability in this scenario is given by:

$$\begin{aligned} \mathrm P \ = \mathrm \ T_0^4 {\mu }^4 e^{\frac{-8 {\pi }^2}{3 \lambda _\mathrm{eff}(\mu )}} \, , \end{aligned}$$

(79)

where $\mu$ is the scale at which the probability is maximum, i.e. $\frac{\partial P}{\partial \mu }=0$, and $T_0$ is the age of the universe. Using condition $\frac{\partial P}{\partial \mu }=0$ along with $\beta _{\lambda }=0$, we can obtain the expression of $\lambda _\mathrm{eff}$ at different scales:

$$\begin{aligned} \lambda _\mathrm{eff}(\mu ) \ = \ \frac{\lambda _\mathrm{eff}(v)}{1-\frac{3}{2\pi ^2}\log \left( \frac{v}{\mu }\right) \lambda _\mathrm{eff}(v)} \, . \end{aligned}$$

(80)

Now, if we set $P=1$, $T_0=10^{10}$ years and $\mu =v$, where $v \simeq 246$ GeV is the EW VEV in Eq. (79), then $\lambda _\mathrm{eff}(v)$ comes out to be 0.0623. However, if we consider $P<1$ with $T_0=10^{10}$ years, then it will be equivalent to demanding that the tunnelling probability from the first vacuum to the deeper one is greater than $T_0$, and we will obtain the condition for metastability as [94]:

$$\begin{aligned} 0 \ > \ \lambda _\mathrm{eff}(\mu ) \ > rsim \ \frac{-0.065}{1-0.01 \log \left( \frac{v}{\mu }\right) }. \end{aligned}$$

(81)

Fig. 15

Higgs mass $M_h$ vs top quark mass $M_t$ plot in GeV for one generation of $\widetilde{R}_2$ and $\varvec{S}_3$. The green and yellow colours represent stable and metastable regions. The black circles are 1$\sigma$, 2$\sigma$ and 3$\sigma$ contours, with the current experimental values of Higgs mass and top quark mass denoted by dots at the centre

Lastly, if the tunnelling probability from the first minimum to the deeper one is less than the age of the universe, i.e. $\lambda _\mathrm{eff}<0$, then the first minimum will be designated as the unstable region. We know that the SM vacuum lies in the metastable region. However, the presence of leptoquarks will exert extra effects in $\lambda _\mathrm{eff}$, which will alter the metastability of the Higgs vacuum. Different regions with respect to stability, metastability and instability for ${\widetilde{R}}_2$, $\varvec{S}_3$ and one generation of ${\widetilde{R}}_2 + \varvec{S}_3$ are presented in Figs. 13, 14 and 15. We refrain from three generations of ${\widetilde{R}}_2 + \varvec{S}_3$, as it obtains serious constraints from Planck scale perturbativity and stability. We have plotted Higgs mass $M_h$ (in GeV) versus top mass $M_t$ (in GeV) in the above-mentioned figures along with the stable, metastable and unstable regions coloured by green, yellow and red, respectively. The black circles define 1$\sigma$, 2$\sigma$ and 3$\sigma$ contours, with a dot at the centre denoting the current Higgs mass and top mass values [144, 148, 149]. In Fig. 13a, b, the results for one generation and three generations of $\widetilde{R}_2$ are illustrated. For this analysis, $M_h$ is varied between 119 GeV and 135 GeV, whereas $M_t$ is varied from 165 GeV to 185 GeV with fixed $\lambda _h=0.1264$ and at the EW scale. The other quartic couplings $\lambda _2$ and ${\widetilde{\lambda }}_2$ are varied from 0.1 to 0.8. As can be seen, for one generation of ${\widetilde{R}}_2$, only the 3$\sigma$ contour hits metastability while the three-generation scenario resides entirely inside the stable region, as the positive effects of gauge couplings and quartic couplings are very large. Again, the positive contributions forming gauge couplings in the triplet leptoquark case are even higher than in the ${\widetilde{R}}_2$ scenario. Therefore, we obtain the complete stable region with both one generation and three generations of $\varvec{S}_3$, shown in Fig. 14a, b. The positive gauge coupling contributions are higher for the ${\widetilde{R}}_2 +\varvec{S}_3$ case, and hence we obtain a completely stable region for this case as well, see Fig. 15.

5 Phenomenology

In this section, we discuss different experimental bounds on the parameter space of scalar leptoquarks and compare them with the theoretical bounds arising from the demand of perturbativity and stability of the theory up to Planck scale. There are both direct and indirect bounds on leptoquarks. While the indirect limits are obtained using effective four-fermion interactions induced by leptoquarks at various low-energy experiments, the direct ones are drawn from the cross section involving their production (if any) at high energy colliders. B-anomalies in semi-leptonic B decays, lepton flavour non-universality, lepton flavour-violating decays, anomalous magnetic moment of muon and rare kaon decays are a few low-energy phenomena constraining leptoquarks. A comprehensive list containing all the indirect bounds on leptoquarks can be found in the “Indirect Limits for Leptoquarks” section of Ref. [149]. However, most of the indirect limits involve bounds on the product of one diagonal and one off-diagonal Yukawa coupling of the leptoquarks with quarks and leptons [92, 150, 151]. Since this coupling has been considered diagonal in our analysis, those indirect limits are automatically satisfied.

On the other hand, it is well known that leptoquarks coupling to multiple generations of quarks and leptons are capable of inducing flavour-changing neutral currents. For example, non-chiral leptoquarks, which can interact with both left- and right-handed leptons, obtain stringent constraints from muon $g-2$ [153] and the ratio of partial decay rates $(\pi \rightarrow e\nu )/(\pi \rightarrow \mu \nu )$ [154] if they are allowed to interact with multiple generations of quarks and fermions. In our analysis, we neither force any leptoquark to couple to different generations of quarks and leptons, nor work with any non-chiral leptoquark.⁷ Therefore, the constraints arising from flavour-changing neutral currents will be much weaker in our scenarios. It is interesting to mention that the possibilities of larger Yukawa couplings of leptoquarks, i.e. $\mathcal {O}(1)$, are not completely ruled out by the low-energy observables [8, 22, 23, 25, 26, 73].

Now, it is impossible to find any single scalar leptoquark solution to all the flavour anomalies, and therefore combinations of different scalar leptoquarks are essential to take various flavour anomalies into account. For example, leptoquarks $S_1$ and $R_2$ can explain the observed anomalies in $R_{D^{(*)}_{}}$, whereas leptoquark $\varvec{S}_3$ can account for $R_{K^{(*)}_{}}$ anomalies [23]. So, in order to describe both the B-anomalies, one should consider $S_1-\varvec{S}_3$ or $R_2-\varvec{S}_3$ pairs.⁸ In Fig. 16, we depict the constraints on the parameter space of $\varvec{S}_3$ describing $R_{K^{(*)}_{}}$ anomalies, where the yellow and green regions indicate $1\sigma$ and $2\sigma$ allowed ranges [15]. Again, To generate tiny neutrino masses through loops within the framework of leptoquark models, one has to combine $S_1$ or $\varvec{S}_3$ with ${\widetilde{R}}_2$ [29, 155]. Moreover, although non-chiral leptoquarks $S_1$ and $R_2$ can accommodate muon and electron $(g-2)$, the masses of the leptoquarks required for illustrating the experimental values are $\approx 100$ TeV considering the Yukawa couplings under perturbative limit [155]. Therefore, one should consider combinations of $S_1$ & $\varvec{S}_3$, $\widetilde{S}_1$ & $\varvec{S}_3$ or $R_2$ and $\widetilde{R}_2$ mixing through the Higgs field [155]. However, the imposition of various flavour physics constraints along with LHC bounds and $\mu \rightarrow e \gamma$ result suggests that none of these scenarios can accommodate both muon and electron $(g-2)$. Therefore, to gain a complete picture regarding various low-energy observables, study of the bounds on the parameter space of different leptoquarks is indispensable.

Fig. 16

Constraint on Yukawa coupling of $\varvec{S}_3$ as a function of its mass describing $R_{K^{(*)}_{}}$ anomalies [15]. The yellow and green colours indicate $1\sigma$ and $2\sigma$ allowed regions

Fig. 17

Flavour constraint on Yukawa coupling of first-generation $\varvec{S}_3$ as a function of its mass [152]. The blue and red lines indicate the bounds from the ratio ${\mathcal {B}} (K^+\rightarrow \pi ^+ ee)/{\mathcal {B}} (K^+\rightarrow \pi ^+ \mu \mu )$ and the branching fraction ${\mathcal {B}} (K^+\rightarrow \pi ^+ \nu \nu )$. The cyan and orange lines indicate constraints from neutral kaon and D-meson mixing, respectively. The black line signifies leptoquark ($\varvec{S}_3$) contribution of − 0.0005 to the Wilson coefficient $(C_{11}^{e\nu })$ involved in the ratio ${\mathcal {B}}(\pi \rightarrow \mu \nu )/{\mathcal {B}}(\pi \rightarrow e\nu )$

Fig. 18

Leading-order Feynman diagrams involving direct bounds on leptoquarks. The first two rows correspond to Leptoquark pair production (PP) at LHC, while the third and fourth rows signify single production (SP) of leptoquark associated with a quark, leptoquark contribution to Drell–Yan-like di-lepton process and single resonant production of leptoquark (SRP). Regarding pair production, the first three diagrams indicate gluon fusion (GF), while the last two illustrate quark fusion (QF). The photon and Z-mediated diagrams have been ignored due to the very small contribution

Fig. 19

Bounds on parameter space of scalar leptoquark coupling to first generation of quarks and leptons [156, 157]. The shaded regions are disallowed by direct detection. The limits from pair production involving charged leptons are recast from [158], while the same involving neutrinos are recast from [90]. These bounds are shown in yellow. The limits emerging from single production (SP), Drell–Yan (DY) and single resonant production (SRP), shown by bluish, maroonish purple and reddish portions, are based on Refs. [157, 159, 160]. On the other hand, the mono-jet limits (bluish) are drawn from Ref. [161]. The dotted lines with magenta and sea green colours represent constraints from weak hypercharge measurements involving $\widetilde{R}_2$ and $\varvec{S}_3$, respectively. Finally, the dashed lines indicate theoretical upper bounds on the Yukawa coupling appearing from Planck scale stability up to two-loop order; the brown and red lines represent the limits for three generations of $\varvec{S}_3$ and $\widetilde{R}_2$, whereas the green and black lines portray the same for one generation of $\varvec{S}_3$ and $\widetilde{R}_2+\varvec{S}_3$

We have already mentioned that we have considered only diagonal Yukawa couplings, whereas most of the indirect bounds involve off-diagonal elements as well. For instance, Fig. 16 shows the bound on $\sqrt{|Y_3^{*32} Y_3^{22}|}$ as a function of mass for $\varvec{S}_3$ to explain $R_{K^{(*)}_{}}$ anomalies. Now, the upper limits on the diagonal Yukawa couplings derived from the demand of Planck scale stability and perturbativity are not expected to change much with the introduction of small off-diagonal couplings. However, these small off-diagonal couplings along with large diagonal elements can now be used to explain various flavour anomalies with respect to different indirect bounds. Again, there arise some additional flavour constraints on the parameter space of the first-generation scalar triplet leptoquark $(\varvec{S}_3)$ [152], which are depicted in Fig. 17, but such bounds do not appear for $\widetilde{R}_2$. Moreover, different low-energy bounds on the Yukawa couplings of the $\widetilde{R}_2-\varvec{S}_3$ model are described in Ref. [29]. However, we are mostly interested in the constraints from the collider perspective.

Fig. 20

Bounds on parameter space of scalar leptoquark coupling to second generation of quarks and leptons [156, 157]. The shaded regions are disallowed by direct detection. The limits from pair production involving charged leptons are recast from [162], while the same involving neutrinos are recast from [90]. These bounds are shown in yellow. The limits emerging from single production (SP), Drell–Yan (DY) and single resonant production (SRP), shown by bluish, maroonish purple and reddish portions, are based on Refs. [157, 159, 160]. On the other hand, the mono-jet limits (bluish) are drawn from Ref. [161]. Finally, the dashed lines indicate theoretical upper bounds on the Yukawa coupling appearing from Planck scale stability up to two-loop order; the colour codes are as mentioned in Fig. 19

Fig. 21

Bounds on parameter space of scalar leptoquark coupling to third generation of quarks and leptons [91, 156]. The shaded regions are disallowed by direct detection. The limit from pair production involving charged leptons is recast from [163], which is shown in yellow. The limits emerging from single production (SP) and Drell–Yan (DY), shown by bluish and maroonish purple portions, are based on Refs. [164, 165]. Finally, the dashed lines indicate theoretical upper bounds on the Yukawa coupling appearing from Planck scale stability up to two-loop order; the colour codes are as mentioned in Fig. 19

When discussing the direct bounds on leptoquarks, we consider pair production (PP), single production (SP) associated with a quark, Drell–Yan processes (DY) and single resonant production of leptoquark (SRP). At the pp collider, like LHC, pair production of leptoquarks can occur through both gluon fusion (GF) and quark fusion (QF), whose corresponding Feynman diagrams are shown in the first and second rows of Fig. 18. The Feynman diagrams for single production of the leptoquark, contribution to Drell–Yan like dilepton processes and SRP are presented in the third and fourth rows of Fig. 18. Regarding the coupling of leptoquarks to charged leptons, we obtain the opposite-sign di-lepton (OSD) signature for DY processes, as shown in Fig 18(h), whereas PP and SP provide di-jet plus OSD and mono-jet plus OSD final states at the detector [43, 73]. Conversely, for leptoquarks coupling to neutrinos, we have di-jet plus missing energy and mono-jet plus missing energy signatures only. The full data set collected at HERA in ep collision excluded the first generation of leptoquarks with mass up to 800 GeV at 95% confidence limits for coupling to be 0.3 [82]. In a more recent study they have modified $Y_\phi /m_\phi$ limits for the first generation of leptoquarks [83]. The CMS collaboration at the LHC also searched for single production of leptoquarks which probe the high coupling region of leptoquarks [88, 160].

We depict different direct constraints on the parameter space of scalar leptoquarks in Figs. 19, 20 and 21. These bounds can be recast for different models of scalar leptoquarks depending on the cross sections and the corresponding decay branching fractions leading to the final states. Figure 19 summarizes the bounds for the first generation of leptoquarks, and Figs. 20 and 21 show the same for the second and third generations of leptoquarks, respectively. All the plots presented in Figs. 19 and 20 are taken from Refs. [156, 157], which use Refs. [90, 158, 162] for PP, Ref. [159] for SP, Ref. [160] for DY, LHC Run II data for SRP and Ref. [161] for mono-jet signature with first and second generations of leptons to restrict the parameter space for leptoquark-quark-lepton coupling below 3.0 with leptoquark mass below 3 TeV. Conversely, Fig. 21a describing constraints on $\phi \tau b$ coupling is taken from Ref. [156], which use Refs. [163,164,165] for their analysis, and Fig. 21b illustrating limits on $\phi \nu b$ coupling is taken from Ref. [91]. For the final states involving charged leptons, the yellow, blueish, maroonish purple and reddish portions indicate the prohibited region from PP, SP, DY and SRP processes, respectively. On the contrary, for the final states involving missing energy, the yellow and bluish regions signify PP and mono-jet signals.

We impose the theoretical bounds obtained from the perturbative unitarity and the stability at the two-loop for the dimensionless couplings in the $m_{\phi }-Y_{\phi }$ plane for $\widetilde{R}_2$, $\varvec{S}_3$ and $\widetilde{R}_2+\varvec{S}_3$, respectively. The brown $(Y_\phi =3.90)$ and the red $(Y_\phi =1.36)$ dashed lines depict the theoretical upper limits on the Yukawa couplings of leptoquarks for three generations of $\varvec{S}_3$ and $\widetilde{R}_2$, respectively, considering Planck scale stability at the two-loop level. The same for one generation of $S_3$ and $\widetilde{R}_2+\varvec{S}_3$ are presented by the green $(Y_\phi =1.29)$ and the black $(Y_\phi =1.21)$ dashed lines.

At this point it is worth mentioning that we do not present the bounds on $\widetilde{R}_2$ with one generation and $\widetilde{R}_2+\varvec{S}_3$ with three generations in these plots. Actually, as described earlier, $\widetilde{R}_2$ with one generation cannot achieve Planck scale stability for any small value of $Y_\phi$ at two-loop order. On the other hand, although $\widetilde{R}_2+\varvec{S}_3$ with three generations shows stability for $Y_\phi \le 1.0$, it loses perturbativity at an energy scale ($\sim$ $10^{14.4}$ GeV) far below the Planck scale at two-loop order.

For the first-generation leptoquark coupling to charged leptons, there exists another bound from measurement of the weak charge of proton and nuclei [156]. This quantity is measured through atomic parity violation and parity-violating electron scattering [149, 166]. For ${\widetilde{R}}_2$ and $\varvec{S}_3$, these measurements translate into $Y_\phi \le 0.17 \bigg (\frac{m_\phi }{1 \mathrm {TeV}}\bigg )$ and $Y_\phi \le 0.21 \bigg (\frac{m_\phi }{1 \mathrm {TeV}}\bigg )$, respectively, which are shown by the dotted lines in magenta and sea green colours, respectively. Since $\widetilde{R}_2$ couples to the down-type quarks only, while $\varvec{S}_3$ interacts with both up-type and down-type quarks, we find the magenta line in Fig. 19b only, whereas the sea green line exists in both Fig. 19a and b. Because the nuclei do not contain other generations of quarks as valance quarks, this kind of limit does not appear for other generations of leptoquarks.

From these results it is evident that the theoretical limits coming from Planck scale stability and perturbative unitarity up to two-loop order might put stronger constraints on the parameter space of leptoquarks with higher mass range in particular for the second and third generations of the leptoquarks. On the other hand, bounds on the Higgs-leptoquark quartic coupling are not very well studied in the literature. In our analysis, we find that this coupling being larger than $\sim 0.2$ disturbs the perturbativity of the theory up to Planck scale.⁹

6 Conclusion

In this paper, we have studied the scalar doublet leptoquark $\widetilde{R}_2$, the scalar triplet leptoquark $\varvec{S}_3$ and their combination with both one generation and three generations with respect to the perturbativity and the stability of the Higgs vacuum. The extra contribution in the running of the gauge couplings at one-loop level mainly depends on the number of the leptoquark components present in the model, which is determined by its gauge structure. Although at the two-loop level they depend on the leptoquark Yukawa couplings, they do not depend on the Higgs-leptoquark couplings explicitly. With the two-loop effects, the gauge coupling $g_2$ for the leptoquark $\varvec{S}_3$ and the combined scenario of $\widetilde{R}_2$ and $\varvec{S}_3$ with three generations diverges at $10^{19.7}$ GeV and $10^{14.4}$ GeV, respectively, which forces the other couplings to hit singularity at those scales. But at the one-loop level, all the leptoquark models considered in this paper achieve Planck scale perturbativity with gauge couplings. It is also noteworthy that no Landau pole emerges in the running of gauge couplings for two generations of these leptoquarks. The Higgs-leptoquark quartic couplings acquire severe constraints from Planck scale perturbativity. With larger EW values of these couplings (e.g. 0.3), the theories become non-perturbative at much lower energy scales than the Planck scale. These constraints do not change much due to alteration in the leptoquark Yukawa couplings. For the three-generation scenario with ${\widetilde{R}}_2$ and $\varvec{S}_3$ combined, the Higgs-leptoquark quartic couplings diverge much below the Planck scale. On the other hand, the leptoquark Yukawa couplings obtain the upper bound from the Planck scale perturbativity and stability of the Higgs vacuum. In the running of $\lambda _{h}$, the gauge couplings exert positive contributions, whereas the Yukawa couplings of leptoquarks introduce negative effects. For three generations of $\widetilde{R}_2$ with the Higgs-leptoquark quartic couplings being 0.1, the Yukawa coupling should be smaller than 1.36 for the theory maintaining stability up to Planck scale. This number becomes 1.29, 3.9¹⁰ and 1.21 for one generation of $\varvec{S}_3$, three generations of $\varvec{S}_3$ and one generation of $\widetilde{R}_2+\varvec{S}_3$, respectively. Finally, regarding the Coleman–Weinberg effective potential approach, the presence of any of these leptoquarks with any number of generations pushes the metastable vacuum of SM to the stable region, although the $3\sigma$ contour of $\widetilde{R}_2$ with one generation marginally touches the metastable region. The phenomenological bounds obtained mainly from the collider experiments are also drawn along with our theoretical bounds. We see that the Planck scale perturbativity and stability puts some theoretical additional restrictions on the parameter space of the leptoquarks in addition to the experimental bounds.

Appendices

Appendix A: Two-loop beta functions of $g_3$

Using SARAH, we generate the beta function of $g_3$ for different models up to two loops, which are given below:

(82)

(83)

(84)

(85)

(86)

(87)

(88)

Appendix B: Two-loop beta functions of $g_1$

Now, with the help of SARAH, we show the two-loops beta function of $g_1$ for all the models as follows:

(89)

(90)

(91)

(92)

(93)

(94)

(95)

Appendix C: Running of top Yukawa coupling

Top Yukawa coupling plays a very important role in the stability of the Higgs vacuum. Therefore, it is important to study the RG evolution of this parameter. As already mentioned in Eq. (15), the absolute value for top Yukawa coupling at any energy scale must be less than $\sqrt{4\pi }$ in order to ensure the perturbativity of the model. It is worth mentioning that although the Yukawa couplings for leptons and other quarks also vary with the scale, their initial value at the EW scale are so small that they usually never cross the perturbativity bound unless some other parameter hits the divergence. Therefore, we restrict our discussion for the top Yukawa coupling only. Now, to investigate the running of the top Yukawa coupling, we study the RG evolution of (Yukawa matrix for up-type quarks) whose (3,3) component would provide us the desired result. The one-loop and two-loop beta functions of under SM are as follows:

(96)

(97)

The above two expressions are matrix equations, with $\text {I}_3$ indicating the $3\times 3$ identity matrix in the flavour space of up-type quarks.

Now, for leptoquark $\widetilde{R}_2$ (with both one generation and three generations), the one-loop beta function of does not obtain any additional contribution at the one-loop level, i.e. . Hence, it remains the same as that of SM:

(98)

Fig. 22

Variation in top quark Yukawa with scale

Nevertheless, there exist some non-vanishing two-loop contributions to it, and hence the full two-loop beta functions of for leptoquark $\widetilde{R}_2$ with one both generation and three generations can be expressed as follows:¹¹

(99)

In the case of leptoquark $\varvec{S}_3$, the correction to the one-loop beta function of contains only one term, and hence it looks like:

(100)

The full two-loop beta function for in this scenario becomes:

(101)

Now, in the combined scenario of $\widetilde{R}_2$ and $\varvec{S}_3$, apart from the individual contributions of $\widetilde{R}_2$ and $\varvec{S}_3$ to the running of , there emerges another at the two-loop level which contains effects of $Y_2$ and $Y_3$ simultaneously. Therefore, the beta function for up to two-loop order in this case can be expressed as:

(102)

(103)

At this point, it is important to mention that for our choice of leptoquark couplings in the three-generation cases, $Y_{\gamma ,i}Y_{\gamma ^\prime ,j}=0$ for $i\ne j$, where $(\gamma ,\gamma ^\prime )\in \{2,3\}$.

We depict the results for the variation in the top Yukawa coupling with the energy scale at the two-loop level under different models in Fig. 22, where the left and right panels show the leptoquark coupling with quarks and leptons ($Y_\phi$) to be 0.4 and 1.0, respectively. The green curve explains the SM scenario, while the yellow and blue lines illustrate $\widetilde{R}_2$ and $\varvec{S}_3$ leptoquarks with three generations. As expected, the SM value of the top Yukawa coupling decreases with energy. With the inclusion of leptoquarks, this coupling shifts further down, and for the case of $\varvec{S}_3$ it achieves divergence at $10^{19.7}$ GeV. Since $\widetilde{R}_2$ and $\varvec{S}_3$ with one generation do not show any abnormal behaviour, we do not present them here. On the other hand, the brown (solid) and black (dashed) curves represent the combined models of $\widetilde{R}_2$ and $\varvec{S}_3$ with one and three generations, respectively. As anticipated, the case with three generations of both leptoquarks stays at the bottom of all the other lines for lower values of $Y_\phi$ (e.g. $Y_\phi =0.4$, shown by the left panel of Fig. 22), although only this curve is noticeably affected if $Y_\phi$ is increased to some sufficiently higher value (e.g. $Y_\phi =1.0$, as exhibited in the right panel of Fig. 22). Like all the other couplings for this scenario, it also diverges at $10^{14.4}$ GeV. The relative positions of the curves in the above-mentioned plot depend primarily on the negative contributions from the gauge couplings at the one-loop level, see Eqs. (96)–(100). With the increase in the number of leptoquark components, the values of gauge couplings are enhanced at any particular energy scale, which in turn will push the top Yukawa coupling downward.

Appendix D: Running of leptoquark Yukawa couplings

Now, let us discuss the evolution of leptoquark Yukawa couplings $Y_2$ and $Y_3$. As mentioned in Eq. (15), these parameters should also have an upper bound of $\sqrt{4\pi }$ at any energy scale $\mu$. For the ${\widetilde{R}}_2$ scenario, the one-loop and two-loop beta functions for $Y_2$ are given by:

(104)

(105)

Now, for three generations of ${\widetilde{R}}_2$, we have three Yukawa matrices of leptoquark ($Y_{2,i}$) corresponding to three different generations of quarks and leptons. The running of each of these Yukawa matrices at the one-loop level can be expressed as:

(106)

At this point we remind the reader again that for any parameter indicates the beta function of that parameter with the replacement of to with $\gamma \in \{2,3\}$ and i representing the generation. It is interesting to note that there appear some additional terms with inter-generation interactions. The beta function for the i-th generation of $Y_{2}$ at two-loop order is given by:

(107)

The term within the curly brackets is added to the above expression in order to incorporate the extra contribution at one-loop order coming from the cross-generation interaction, as shown in Eq. (106). The rest of the terms arise from two-loop contributions.

In a similar fashion, the one-loop and two-loop beta functions for $Y_3$ in the case of leptoquark $\varvec{S}_3$ can be expressed as follows:

(108)

(109)

For the three-generation case of $\varvec{S}_3$, the one-loop beta of the i-th-generation leptoquark Yukawa coupling takes the form:

(110)

As in the case of ${\widetilde{R}}_2$, here also some new terms appear due to inter-generation interactions at the one-loop level. Encompassing these additional terms, the two-loop beta function can be written as:

(111)

Fig. 23

Variation in leptoquark Yukawa with scale

Now, for the combined scenario of ${\widetilde{R}}_2$ and $\varvec{S}_3$, the above expressions are modified, and extra contributions from the interactions of doublet and triplet leptoquarks emerge at both the one- and two-loop levels. Thus, they can be written as follows:

(112)

(113)

(114)

(115)

In all of the above four expressions, the first term indicates the extra contribution at one-loop order due to the presence of both doublet and triplet leptoquarks.

Since one-generation cases do not show any irregularities, we depict the variations in $Y_2$ and $Y_3$ in three-generation scenarios of the leptoquarks in Fig. 23. Figure 23a, b in the first row illustrate the variations in any diagonal element of $Y_2$ starting from 0.4 and 1.0, respectively, while Fig. 23c, d in the second row demonstrate the similar situation for $Y_3$. As can be observed, for low Yukawa, the combined scenarios stay below the individual cases, whereas the situation flips for higher Yukawa cases due to the large effects from the combined terms of $Y_2$ and $Y_3$. As expected, $Y_3$ for three generations of $\varvec{S}_3$ decreases monotonically with energy and hits the divergence at $10^{19.7}$ GeV, while both $Y_2$ and $Y_3$ diverge at $10^{14.4}$ GeV for the three-generation $\widetilde{R}_2+\varvec{S}_3$ case. However, $Y_2$ shows different behaviour for large Yukawa. For the $\widetilde{R}_2$ case, as can be observed from Fig. 23b, it initially decreases with scale, then reaches a minimum and gradually starts increasing. For the $\widetilde{R}_2+\varvec{S}_3$ case, it increases with energy from the beginning, then reaches a maximum and starts to fall off, but suddenly blows up at $10^{14.4}$ GeV.

Appendix E: Two-loop beta functions of Higgs-leptoquark quartic couplings for $\widetilde{R}_2$

(116)

(117)

(118)

(119)

Appendix F: Two-loop beta functions of Higgs-leptoquark quartic couplings for $\varvec{S}_3$

(120)

(121)

(122)

(123)

Appendix G: Two-loop beta functions of Higgs-leptoquark quartic couplings for $\widetilde{R}_2+\varvec{S}_3$

(124)

(125)

(126)

(127)

(128)

(129)

(130)

(131)