Bottom quark and tau lepton masses in a toy $\textrm{SU}(6)$ model

Abstract

We study a toy $\textrm{SU}(6)$ model with the symmetry breaking pattern of the extended 331 symmetry of $\textrm{SU}(3)_c \otimes \textrm{SU}(3)_W \otimes \textrm{U}(1)_X$. A “fermion-Higgs mismatching” symmetry breaking pattern is proposed for more realistic model building. Within such symmetry breaking pattern, only one Higgs doublet develops vacuum expectation value for the spontaneous electroweak symmetry breaking, and gives tree-level top quark mass. A natural VEV splittings in the 331 breaking Higgs fields gives tree-level masses to both bottom quark and tau lepton. The $125\,\textrm{GeV}$ SM-like Higgs boson discovered at the LHC can have Yukawa couplings to bottom quark and tau lepton as in the SM prediction, and this suggests the 331 symmetry breaking scale to be $\sim {{{\mathcal {O}}}}(10)\,\textrm{TeV}$.

1 Introduction

Grand Unified Theories (GUTs) [1, 2] were proposed to unify all fundamental interactions and elementary particles described by the Standard Model (SM) at the electroweak (EW) scale. Meanwhile, a unified description of the generational structure as well as the SM fermion mass hierarchies have not been realized in terms of the $\textrm{SU}(5)$ or $\textrm{SO}(10)$ GUTs. This is largely due to the fact that three generations of SM fermions are accommodated in the $\textrm{SU}(5)$ or $\textrm{SO}(10)$ GUTs by simple repetition of one anomaly-free fermion generation. Consequently, the symmetry breaking patterns do not provide any source for the observed SM fermion mass hierarchies. It was pointed out and discussed in Refs. [3,4,5,6,7] that multiple fermion generations, such as $n_g=3$ for the SM case, can be embedded non-trivially in GUT groups of $\textrm{SU}(7)$ and beyond.¹ Therefore, it is natural to conjecture that the SM fermion mass hierarchies may originate from the intermediate symmetry breaking scale of some non-minimal GUT with $\textrm{SU}(N\ge 7)$ [4, 5]. Historically, the embedding of the SM generations as well as fermion mass hierarchies were studied in the context of technicolor and extended technicolor models [8,9,10,11,12], where the symmetry breakings are due to the fermion bi-linear condensates. Given the discovery of a single $125\,\textrm{GeV}$ SM-like Higgs boson at the Large Hadron Collider (LHC) [13, 14] until now, it is pragmatic to revisit the flavor issue in the framework of GUTs, where the spontaneous symmetry breakings are achieved by the Higgs mechanism.

Besides of addressing the flavor puzzle, it was also pointed out that the non-minimal GUTs can automatically give rise to the global Peccei–Quinn (PQ) symmetry [16] for the strong CP problem.² This is due to the emergent global symmetry of $\textrm{SU}(N)\otimes \textrm{U}(1)$ in the rank-2 anti-symmetric $\textrm{SU}(N+4)$ gauge theories (with $N\ge 2$), which was first pointed out by Dimopoulos, Raby, and Susskind [17]. In this regard, the longstanding flavor puzzle as well as the PQ quality problem [18,19,20] may be simultaneously addressed within the non-minimal GUTs.³

Before the ambitious goal of understanding the known SM fermion mass hierarchies in realistic non-minimal GUTs, it will be useful to ask whether the minimal version of this class already had some general properties in producing the SM fermion masses. Among various non-minimal GUTs with $\textrm{SU}(N\ge 7)$, indeed, an extended gauge symmetry of ${\mathcal {G}}_{331}\equiv \textrm{SU}(3)_c \otimes \textrm{SU}(3)_W \otimes \textrm{U}(1)_X$ above the EW scale is usually predicted.⁴ This class of models are collectively known as the 331 model, and were previously studied in Refs. [22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50]. This motivates us to consider the $\textrm{SU}(6)$ as a one-generational toy model,⁵ which can be spontaneously broken to ${\mathcal {G}}_{331}$ by its adjoint Higgs field of $\mathbf {35_H}$. An advantage of considering the one-generational $\textrm{SU}(6)$ instead of the 331 model is that one can uniquely define the electric charges for both fermions and gauge bosons in the spectrum. Meanwhile, the previous studies based on the 331 model itself often allowed different charge quantization schemes [24, 37, 39, 44, 48], which could potentially lead to fermions with exotic electric charges.

After the GUT symmetry breaking, there can be three $\textrm{SU}(3)_W$ anti-fundamental Higgs fields in the 331 model. In the previous studies, only one of them developed a vacuum expectation value (VEV) of $V_{331}$ for the symmetry breaking of $\textrm{SU}(3)_W \otimes \textrm{U}(1)_X \rightarrow \textrm{SU}(2)_W \otimes \textrm{U}(1)_Y$, while two others developed VEVs of $v_{\textrm{EW}} \simeq 246\,\textrm{GeV}$ for the electroweak symmetry breaking (EWSB). According to the Yukawa couplings, one can identify a type-II two-Higgs-doublet model (2HDM) at the EW scale for the 331 model. By extending to larger non-minimal GUTs for $n_g=3$, such as $\textrm{SU}(9)$ as our example, the conventional symmetry breaking pattern in the 331 model predicts more than two EW Higgs doublets. This is certainly very problematic given that the direct searches for the second Higgs doublet at the Large Hadron Collider (LHC) give null results so far. Motivated by the general features in the Higgs sector of the non-minimal GUT, we study the alternative symmetry breaking pattern with only one EW Higgs doublet coming from the $\mathbf {15_H}$ of the $\textrm{SU}(6)$. An immediate question is how do bottom quark and tau lepton acquire masses given the vanishing tree-level Yukawa couplings. It turns out their masses can only be obtained when two $\textrm{SU}(3)_W$ anti-fundamental Higgs fields from $\mathbf {\overline{6_H}}^{\rho =1,2}$ develop VEVs both for the 331 and EW symmetry breaking directions. A natural mass splitting between the top quark and the $(b,\tau )$ in the third generation can be achieved with ${\mathcal {O}}(1)$ Yukawa couplings. The corresponding 331 breaking scale is found to be $V_{331} \sim {\mathcal {O}}(10)\,\textrm{TeV}$ from the Yukawa couplings of the SM-like Higgs boson with the $(b,\tau )$. Historically, a universal ${\mathcal {O}}(1)$ Yukawa coupling was motivated by observing the natural top quark mass at the EW scale, and this was generalized as the anarchical fermion mass scenario in the studies of the neutrino masses [51, 52]. We also wish to remind the readers, the whole discussions are based on the 331 model due to the minimal one-generational $\textrm{SU}(6)$ symmetry breaking. Aside from the SM fermion masses, we do not address some general questions of gauge coupling unification or proton lifetime predictions. Neither do we determine whether a supersymmetric extension to the current model is necessary, with the belief that this would better be studied in more realistic models with $n_g=3$. Some related discussions can be found in Refs. [41, 43, 46].

The rest of the paper is organized as follows. In Sect. 2, we motivate the possible symmetry breaking pattern from two independent aspects in the toy $\textrm{SU}(6)$ model, which leads to only one Higgs doublet for the spontaneous EWSB. In Sect. 3, we describe the Higgs sector in the $\textrm{SU}(6)$ GUT, with the emphasis on the reasonable mass generations to the bottom quark and tau lepton through the Yukawa couplings. In Sect. 4, we describe the bottom quark and tau lepton masses in the toy $\textrm{SU}(6)$ model based on the reasonable symmetry breaking pattern as well as the VEV assignment. Some comments will be made for the necessary condition of the radiative mass generation in the current context. We summarize our results and make discussions in Sect. 5. An Appendix A is provided to summarize the gauge sector as well as the fermion Yukawa couplings of the 331 model. All Lie group calculations in this work are carried out by LieART [53, 54].

2 One-generational $\textrm{SU}(6)$

The minimal anomaly-free $\textrm{SU}(6)$ GUT contains the left-handed fermions of

$$\begin{aligned} \{ f_L \}_{ \textrm{SU}(6) }= & {} \mathbf {\overline{6_F}}^\rho \oplus \mathbf {15_F},\quad \rho =1,2. \end{aligned}$$

(1)

The fermion sector enjoys a global symmetry of

$$\begin{aligned} {{{\mathcal {G}}}}_{\textrm{flavor}}= & {} \textrm{SU}(2)_F \otimes \textrm{U}(1)_{\textrm{PQ}}, \end{aligned}$$

(2)

according to Ref. [17]. The most general Yukawa couplings that are invariant under the gauge symmetry are expressed as⁶

$$\begin{aligned} {-}{\mathcal {L}}_Y= & {} (Y_{\mathcal {D}})_{\rho \sigma } \mathbf {15_F} \mathbf {\overline{6_F}}^\rho \mathbf {\overline{6_H}}^\sigma {+} Y_{\mathcal {U}}\mathbf {15_F} \mathbf {15_F} \mathbf {15_H} {+} H.c., \end{aligned}$$

(3)

where we allow the explicit $\textrm{SU}(2)_F$-breaking term in the Yukawa couplings, so that $(Y_{\mathcal {D}})_{\rho \sigma } \ne Y_{\mathcal {D}}\delta _{\rho \sigma }$.

Below, we motivate our Higgs VEV assignments for the viable symmetry breaking from three different aspects, which include

${\mathfrak {A}}$:

the null results in searching for a second Higgs doublet at the LHC,

${\mathfrak {B}}$:

the extension to the non-minimal GUTs with $n_g=3$, e.g., the $\textrm{SU}(8)$ GUT [6, 7],

${\mathfrak {C}}$:

the natural mass generation of the bottom quark and tau lepton with Yukawa couplings of $\sim {\mathcal {O}}(1)$.

2.1 The symmetry breaking pattern

The viable $\textrm{SU}(6)$ breaking pattern is expected to be

$$\begin{aligned}{} & {} \textrm{SU}(6) \xrightarrow {\Lambda _{\textrm{GUT}} } {{{\mathcal {G}}}}_{331} \xrightarrow { V_{331}} {{{\mathcal {G}}}}_{\textrm{SM}},\nonumber \\{} & {} {{{\mathcal {G}}}}_{331} = \textrm{SU}(3)_c \otimes \textrm{SU}(3)_W \otimes \textrm{U}(1)_X,\nonumber \\{} & {} {{{\mathcal {G}}}}_{\textrm{SM}}= \textrm{SU}(3)_c \otimes \textrm{SU}(2)_W \otimes \textrm{U}(1)_Y, \end{aligned}$$

(4)

where the GUT scale symmetry breaking is achieved by an $\textrm{SU}(6)$ adjoint Higgs field of $\mathbf {35_H}$. The $\textrm{U}(1)_X$ charge for the $\textbf{6} \in \textrm{SU}(6)$ and $\textrm{U}(1)_Y$ charge for the $\textbf{3}_W \in \textrm{SU}(3)_W$ are defined by

$$\begin{aligned} X( \textbf{6} )\equiv & {} \textrm{diag} \left( -\frac{1}{3}, -\frac{1}{3}, -\frac{1}{3}, +\frac{1}{3}, +\frac{1}{3}, +\frac{1}{3}\right) , \end{aligned}$$

(5a)

$$\begin{aligned} Y( \textbf{3}_W )\equiv & {} \textrm{diag} \left( \frac{1}{6} + X, \frac{1}{6} +X, -\frac{1}{3} + X\right) . \end{aligned}$$

(5b)

The electric charge operator of the $\textrm{SU}(3)_W$ fundamental representation is expressed as a $3\times 3$ diagonal matrix

$$\begin{aligned} Q ( \textbf{3}_W )\equiv & {} T_{\textrm{SU}(3)}^3 {+} Y = \textrm{diag} \left( \frac{2}{3}{+}X,{-} \frac{1}{3}{+}X, {-}\frac{1}{3}{+}X \right) . \end{aligned}$$

(6)

with the first $\textrm{SU}(3)$ Cartan generator of

$$\begin{aligned} T_{\textrm{SU}(3)}^3= & {} \frac{1}{2} \textrm{diag} (1,-1,0 ). \end{aligned}$$

(7)

Accordingly, we find that Higgs fields in Eq. (3) are decomposed as follows

$$\begin{aligned}{} & {} \mathbf {\overline{6_H}}^\rho = \left( \mathbf {\overline{3}},\textbf{1}, +\frac{1}{3} \right) _{ {\textbf{H}}}^\rho \oplus \overbrace{ \left( \textbf{1},\mathbf {\overline{3}}, -\frac{1}{3} \right) _{ {\textbf{H}} }^\rho }^{ \mathbf {\Phi }_{ \mathbf {\overline{3}}, \rho } },\nonumber \\{} & {} \mathbf {15_H} = \left( \mathbf {\overline{3}},\textbf{1}, -\frac{2}{3} \right) _{ {\textbf{H}}} \oplus \overbrace{ \left( \textbf{1},\mathbf {\overline{3}}, +\frac{2}{3} \right) _{ {\textbf{H}}} }^{ \mathbf {\Phi }_{\mathbf {\overline{3}}}^\prime } \oplus ( \textbf{3},\textbf{3}, 0 )_{ {\textbf{H}}}, \end{aligned}$$

(8)

for the symmetry breaking pattern in Eq. (4). Two $( \textbf{ 1},\! \mathbf {\overline{3}},\! -\frac{1}{3})_{ {\textbf{H}}}^\rho \subset \mathbf {\overline{6_H}}^\rho$ contain SM-singlet directions after the second stage symmetry breaking in Eq. (4). Meanwhile, the $( \textbf{ 1} , \mathbf {\overline{2}} , +\frac{1}{2} )_{ {\textbf{H}}} \subset ( \textbf{ 1} , \mathbf {\overline{3}} , +\frac{2}{3})_{ {\textbf{H}}} \subset \mathbf {15_H}$ can only develop VEV to trigger the spontaneous EWSB of $\textrm{SU}(2)_W \otimes \textrm{U}(1)_Y \rightarrow \textrm{U}(1)_{\textrm{em}}$. Under the symmetry breaking pattern in Eq. (4) and the charge quantization given in Eqs. (5a), (5b), and (6), we summarize the $\textrm{SU}(6)$ fermions and their names in Table 1. For the SM fermions marked by solid underlines, we name them by the third generational SM fermions. This will become manifest from their mass origin within the current context.

In previous studies of the 331 model [22, 24,25,26,27,28,29,30,31,32,33,34,35,36,37,38, 45,46,47,48,49], there were two Higgs doublets at the EW scale, which come from the $\mathbf {\overline{6_H}}^1$ (chosen to be $\rho =1$ without loss of generality) and the $\mathbf {15_H}$, respectively. Schematically, one expresses the VEVs of $\textrm{SU}(3)$ anti-fundamentals as follows

$$\begin{aligned}{} & {} \left\langle \left( \textbf{1},\mathbf {\overline{3}}, -\frac{1}{3} \right) _{ {\textbf{H}}}^1 \right\rangle = \frac{1}{ \sqrt{2}} \left( \begin{array}{c} 0 \\ v_d \\ 0 \\ \end{array} \right) ,\quad \left\langle \left( \textbf{1},\mathbf {\overline{3}}, -\frac{1}{3} \right) _{ {\textbf{H}}}^2 \right\rangle \nonumber \\{} & {} = \frac{1}{ \sqrt{2}} \left( \begin{array}{c} 0 \\ 0 \\ V_{331} \\ \end{array} \right) , \left\langle \left( \textbf{1},\mathbf {\overline{3}}, +\frac{2}{3} \right) _{ {\textbf{H}}} \right\rangle = \frac{1}{ \sqrt{2}} \left( \begin{array}{c} v_u \\ 0 \\ 0 \\ \end{array} \right) , \end{aligned}$$

(9)

with $v_u^2 + v_d^2 = v_{\textrm{EW}}^2 \approx (246\,\textrm{GeV})^2$. In such a scenario, the bottom quark and tau lepton masses are given by Yukawa couplings as follows

$$\begin{aligned}{} & {} \mathbf {15_F} \mathbf {\overline{6_H}}^1 \mathbf {\overline{6_F}}^1 +H.c. \supset \left[ ( \textbf{3},\textbf{3}, 0 )_{ {\textbf{F}}} \otimes \left( \mathbf {\overline{3}},\textbf{1}, +\frac{1}{3 }\right) _{ {\textbf{F}}}^1 \right. \nonumber \\{} & {} \quad \left. \oplus \left( \textbf{1},\mathbf {\overline{3}}, +\frac{2}{3 } \right) _{ {\textbf{F}}} \otimes \left( \textbf{1},\mathbf {\overline{3}}, -\frac{1}{3 } \right) _{ {\textbf{F}}}^1 \right] \nonumber \\{} & {} \quad \otimes \left( \textbf{1},\mathbf {\overline{3}}, -\frac{1}{3 }\right) _{ {\textbf{H}}}^1 + H.c. \nonumber \\{} & {} \quad \supset \left[ \left( \textbf{3},\textbf{2}, +\frac{1}{6}\right) _{ {\textbf{F}}} \otimes \left( \mathbf {\overline{3}},\textbf{1}, +\frac{1}{3 } \right) _{ {\textbf{F}}}^1\right. \nonumber \\{} & {} \quad \left. \oplus \, ( \textbf{1},\textbf{1}, +1)_{ {\textbf{F}}} \otimes \left( \textbf{1},\mathbf {\overline{2}}, -\frac{1}{2 } \right) _{ {\textbf{F}}}^1 \right] \nonumber \\{} & {} \quad \otimes \left( \textbf{1},\mathbf {\overline{2}}, -\frac{1}{2}\right) _{ {\textbf{H}}}^1 + H.c., \end{aligned}$$

(10)

while the top quark mass is given by Yukawa coupling as follows

$$\begin{aligned}{} & {} \mathbf {15_F} \mathbf {15_F} \mathbf {15_H} + H.c. \supset \left( \textbf{3},\textbf{3}, 0 \right) _{ {\textbf{F}}} \otimes \left( \mathbf {\overline{3}},\textbf{1}, -\frac{2}{3 } \right) _{ {\textbf{F}}} \nonumber \\{} & {} \quad \otimes \left( \textbf{1},\mathbf {\overline{3}}, +\frac{2}{3 } \right) _{ {\textbf{H}}} + H.c. \nonumber \\{} & {} \quad \supset \left( \textbf{3},\textbf{2}, +\frac{1}{6} \right) _{ {\textbf{F}}} \otimes \left( \mathbf {\overline{3}},\textbf{1}, -\frac{2}{3} \right) _{ {\textbf{F}}}\nonumber \\{} & {} \quad \otimes \left( \textbf{1},\mathbf {\overline{2}}, +\frac{1}{2 } \right) _{ {\textbf{H}}} + H.c. \end{aligned}$$

(11)

All charged fermion masses in the third generation are due to the spontaneous EWSB. Therefore, the Higgs sector at the EW scale is described by a type-II 2HDM.⁷ Furthermore, the other Yukawa coupling of $\mathbf {15_F} \mathbf {\overline{6_H}}^2 \mathbf {\overline{6_F}}^2 +H.c.$ give fermion masses of $m_B = m_E = m_N \sim {{{\mathcal {O}}}}(V_{331})$ [46], according to the VEV assignment in Eq. (9). By integrating out these massive fermions, the residual massless fermions are found to be anomaly-free under the ${{{\mathcal {G}}}}_{\textrm{SM}}$. Loosely speaking, one anti-fundamental fermion of $\mathbf {\overline{6_F}}^2$ acquires mass with one Higgs field of $( \textbf{1},\mathbf {\overline{3}}, -\frac{1}{3} )_{ {\textbf{H}}}^2 \subset \mathbf {\overline{6_H}}^2$ developing its VEV of ${\mathcal {O}}(V_{331})$.⁸ Thus, we name such symmetry breaking pattern as the “fermion-Higgs matching pattern”.

However, the ongoing probes of the second Higgs doublet at the LHC lack direct evidences for the predicted neutral and charged Higgs bosons from various channels [56,57,58,59,60,61,62,63,64,65,66,67]. In the type-II 2HDM, hierarchical Yukawa couplings of $Y_{\mathcal {U}}\gg Y_{\mathcal {D}}$ can be expected for the third-generational SM fermion masses. As will be shown below, the suppressed $(b,\tau )$ masses can be realized with a more natural Yukawa couplings of $Y_{\mathcal {U}}\sim Y_{\mathcal {D}}\sim {\mathcal {O}}(1)$ in the current context. Aside from the experimental facts, it is most natural to consider the following VEVs for the Higgs fields

$$\begin{aligned}{} & {} \left\langle \left( \textbf{1},\mathbf {\overline{3}}, -\frac{1}{3} \right) _{ {\textbf{H}}}^{\rho =1,2} \right\rangle = \frac{1}{ \sqrt{2}} \left( \begin{array}{c} 0 \\ 0 \\ V_\rho \\ \end{array} \right) \nonumber \\{} & {} \quad \left\langle \left( \textbf{1},\mathbf {\overline{3}}, +\frac{2}{3} \right) _{ {\textbf{H}}} \right\rangle = \frac{1}{ \sqrt{2}} \left( \begin{array}{c} v_u \\ 0 \\ 0 \\ \end{array} \right) , \end{aligned}$$

(12)

purely from the group theoretical point of view. Obviously, two ${\mathcal {G}}_{331}$-breaking VEVs in Eq. (12) are in the SM-singlet components, and one expects the natural hierarchy of $V_1 \sim V_2 \sim {\mathcal {O}}(V_{331} ) \gg v_u = v_{\textrm{EW}}$. By taking the $\textrm{SU}(2)_F$-invariant Yukawa couplings of $(Y_{\mathcal {D}})_{\rho \sigma } = Y_{\mathcal {D}}\delta _{\rho \sigma }$ in Eq. (3) for simplicity, we have the following mass terms

$$\begin{aligned}{} & {} Y_{\mathcal {D}}\mathbf {15_F} \mathbf {\overline{6_H}}^\rho \mathbf {\overline{6_F}}^\rho + H.c. \supset Y_{\mathcal {D}}\left[ ( {\textbf{3}}, {\textbf{3}}, 0 )_{ {\textbf{F}}} \otimes \left( {\textbf{1}}, \bar{ {\textbf{3}}}, -\frac{1}{3} \right) _{ {\textbf{H}}}^\rho \right. \nonumber \\{} & {} \quad \otimes \left( \bar{{\textbf{3}}}, {\textbf{1}}, +\frac{1}{3} \right) _{ {\textbf{F}}}^{ \rho } \oplus \left( {\textbf{1}}, \bar{{\textbf{3}}}, +\frac{2}{3} \right) _{ {\textbf{F}}}\nonumber \\{} & {} \quad \otimes \left. \left( {\textbf{1}}, \bar{ {\textbf{3}}}, -\frac{1}{3} \right) _{ {\textbf{H}}}^\rho \otimes \left( {\textbf{1}}, \bar{{\textbf{3}}}, -\frac{1}{3} \right) _{ {\textbf{F}}}^{\rho } \right] + H.c. \nonumber \\{} & {} \quad \Rightarrow \frac{1}{ \sqrt{2}} ({\bar{b}}_L, {\bar{B}}_L ) \cdot \left( \begin{array}{cc} 0 &{} \quad 0 \\ Y_{\mathcal {D}}V_1 &{}\quad Y_{\mathcal {D}}V_2 \\ \end{array} \right) \cdot \left( \begin{array}{c} b_R \\ B_R \\ \end{array} \right) \nonumber \\{} & {} \qquad +\frac{1}{ \sqrt{2}} ({\bar{\tau }}_L, {\bar{E}}_L ) \cdot \left( \begin{array}{cc} 0 &{}\quad -Y_{\mathcal {D}}V_1 \\ 0 &{}\quad -Y_{\mathcal {D}}V_2 \\ \end{array} \right) \cdot \left( \begin{array}{c} \tau _R \\ E_R \\ \end{array} \right) + H.c.,\nonumber \\ \end{aligned}$$

(13)

with the alternative VEV assignment in Eq. (12). Clearly, both bottom quark and tau lepton remain massless after the spontaneous breaking of the 331 symmetry. Meanwhile, there is still only one anti-fundamental fermion of $\mathbf {\overline{6_F}}^2$ becoming massive. In this regard, the alternative symmetry breaking pattern achieved by both $( \textbf{ 1} , \mathbf {\overline{3}} , -\frac{1}{3})_{ {\textbf{H}}}^\rho \subset \mathbf {\overline{6_H}}^\rho$ is also valid from the anomaly-free condition. Thus, we name the VEV assignment in Eq. (12) as the “fermion-Higgs mismatching pattern” of symmetry breaking. As we shall show below, this VEV assignment leads to a distinct Higgs spectrum from the 2HDM at the EW scale.

2.2 An example: $\textrm{SU}(8)$ with three generations

Besides of the above phenomenological consideration, a better motivation of the current study can be made for non-minimal GUTs with multiple generations. Let us take the $\textrm{SU}(8)$ GUT as an example, which can automatically lead to $n_g=3$ with the following fermion content [6, 7]

$$\begin{aligned} \{ f_L \}_{ \textrm{SU}(8) }= & {} \left[ \mathbf {\overline{8_F}}^\rho \oplus \mathbf {28_F} \right] \bigoplus \left[ \mathbf {\overline{8_F}}^{{\dot{\rho }}} \oplus \mathbf {56_F} \right] , \end{aligned}$$

(14)

according to the rule in Ref. [3]. This setup enjoys an emergent global symmetry of $\left[ \textrm{SU}(4)_1 \otimes \textrm{U}(1)_{1} \right] \otimes \left[ \textrm{SU}(5)_2 \otimes \textrm{U}(1)_{2} \right]$ [8, 21], with the flavor indices of $\rho =1,\ldots ,4$ and ${\dot{\rho }}=5,\ldots ,9$. To focus on the third-generational fermions, we only consider the rank-2 sector in Eq. (14). $\textrm{SU}(4)_1$-invariant Yukawa couplings of $\mathbf {28_F} \mathbf {\overline{8_H}}^\rho \mathbf {\overline{8_F}}^\rho + H.c.$ can be expected to give bottom quark and tau lepton masses, which are analogous to $\mathbf {15_F} \mathbf {\overline{6_H}}^\rho \mathbf {\overline{6_F}}^\rho + H.c.$ in the one-generational $\textrm{SU}(6)$ model. Another gauge-invariant Yukawa coupling of $\mathbf {28_F} \mathbf {28_F} \mathbf {70_H} + H.c.$ is expected to give the top quark mass. A possible symmetry breaking pattern of $\textrm{SU}(8)$ can be expected as follows

$$\begin{aligned}{} & {} \textrm{SU}(8) \xrightarrow {\Lambda _{\textrm{GUT}} } \textrm{SU}(4)_c \otimes \textrm{SU}(4)_W \otimes \textrm{U}(1)_{X_0 } \xrightarrow { V_{441} } \textrm{SU}(3)_c \nonumber \\{} & {} \quad \otimes \textrm{SU}(4)_W \otimes \textrm{U}(1)_{X_1 } \nonumber \\{} & {} \quad \xrightarrow { V_{341} } \textrm{SU}(3)_c \otimes \textrm{SU}(3)_W \otimes \textrm{U}(1)_{X_2} \xrightarrow { V_{331} } \textrm{SU}(3)_c\nonumber \\{} & {} \quad \otimes \textrm{SU}(2)_L \otimes \textrm{U}(1)_Y. \end{aligned}$$

(15)

Here, $(V_{441}, V_{341}, V_{331})$ represent three intermediate symmetry-breaking scales above the EW scale. The corresponding $\textrm{U}(1)$ charges in Eq. (15) are defined by

$$\begin{aligned} X_0(\textbf{8}){} & {} \equiv \textrm{diag} \left( {-} \frac{1}{4}, {-} \frac{1}{4}, {-} \frac{1}{4}, {-} \frac{1}{4},{+}\frac{1}{4}, {+}\frac{1}{4}, {+}\frac{1}{4}, {+}\frac{1}{4} \right) , \end{aligned}$$

(16a)

$$\begin{aligned} X_1 ( \textbf{4}_c ){} & {} \equiv \textrm{diag} \left( {-} \frac{1}{12} {+} X_0, {-} \frac{1}{12} {+} X_0, {-} \frac{1}{12} {+} X_0, \frac{1}{4} {+} X_0 \right) ,\end{aligned}$$

(16b)

$$\begin{aligned} X_2 ( \textbf{4}_W ){} & {} \equiv \textrm{diag} \left( \frac{1}{12} {+} X_1, \frac{1}{12} {+} X_1, \frac{1}{12} {+} X_1, {-}\frac{1}{4} {+} X_1\right) ,\end{aligned}$$

(16c)

$$\begin{aligned} Y ( \textbf{4}_W ){} & {} \equiv \textrm{diag}\left( \frac{1}{4} {+} X_1, \frac{1}{4} {+} X_1, {-}\frac{1}{4} {+}X_1, {-}\frac{1}{4} {+} X_1 \right) . \end{aligned}$$

(16d)

Following the above symmetry breaking pattern and charge quantizations in Eqs. (16), one can decompose the minimal set of Higgs fields as

$$\begin{aligned} \mathbf {\overline{8_H}}^{ \rho }&\supset&\left( \mathbf {\overline{4}}, \textbf{1}, +\frac{1}{4} \right) _{ {\textbf{H}} }^\rho \oplus \left( \textbf{1}, \mathbf {\overline{4}}, -\frac{1}{4} \right) _{ {\textbf{H}} }^\rho ,\end{aligned}$$

(17a)

$$\begin{aligned} \left( \mathbf {\overline{4}}, \textbf{1}, +\frac{1}{4} \right) _{ {\textbf{H}} }^\rho= & {} \left( \mathbf {\overline{3}}, \textbf{1}, +\frac{1}{3} \right) _{ {\textbf{H}} }^\rho \oplus ( \textbf{1}, \textbf{1}, 0 )_{ {\textbf{H}} }^\rho ,\end{aligned}$$

(17b)

$$\begin{aligned} \left( \textbf{1}, \mathbf {\overline{4}}, -\frac{1}{4}\right) _{ {\textbf{H}} }^\rho= & {} \left( \textbf{1}, \mathbf {\overline{3}}, -\frac{1}{3}\right) _{ {\textbf{H}}}^\rho \oplus ( \textbf{1}, \textbf{1}, 0)_{ {\textbf{H}} }^{\rho ^\prime } \nonumber \\= & {} \left( \textbf{1}, \mathbf {\overline{2}}, -\frac{1}{2} \right) _{ {\textbf{H}}}^\rho \oplus ( \textbf{1}, \textbf{1}, 0)_{ {\textbf{H}} }^\rho \oplus ( \textbf{1}, \textbf{1}, 0)_{ {\textbf{H}} }^{\rho ^\prime },\end{aligned}$$

(17c)

$$\begin{aligned} \mathbf {70_H}&\supset&\left( \textbf{4}, \mathbf {\overline{4}}, +\frac{1}{2}\right) _{ {\textbf{H}}} \supset \left( \textbf{1}, \mathbf {\overline{4}}, +\frac{3}{4} \right) _{ {\textbf{H}}} \nonumber \\&\supset&\left( \textbf{1}, \mathbf {\overline{3}}, +\frac{2}{3} \right) _{ {\textbf{H}}}^{\prime \prime \prime } = \left( \textbf{1}, \mathbf {\overline{2}}, +\frac{1}{2} \right) _{ {\textbf{H}}}^{\prime \prime \prime } \oplus ( \textbf{1}, \textbf{1}, +1 )_{ {\textbf{H}}}. \end{aligned}$$

(17d)

Both the $\mathbf {\overline{8_H}}^{ \rho }$ and the $\mathbf {70_H}$ contain the EWSB components of $( \textbf{1} , \mathbf {\overline{2}} , -\frac{1}{2})_{{\textbf{H}}}^\rho$ and $( \textbf{1} , \mathbf {\overline{2}} , +\frac{1}{2})_{{\textbf{H}} }$, respectively. Besides, the four $\mathbf {\overline{8_H}}^{ \rho }$ Higgs fields contain three singlet components for the intermediate symmetry breaking in Eq. (15). A more careful counting by the anomaly-free condition at each stage of symmetry breaking shows that the Higgs spectrum is left with one EW Higgs doublets from the $\mathbf {\overline{8_H}}^{ \rho }$ and one from the $\mathbf {70_H}$ if one adopted the “fermion-Higgs matching pattern” of symmetry breaking, as can be expected for the $n_g=3$ case. In this regard, to have a realistic Higgs spectrum at the EW scale, a “fermion-Higgs mismatching pattern” of the intermediate symmetry breaking can be generally expected.

3 The Higgs sector of the $\textrm{SU}(6)$

In this section, we describe the Higgs sector according to the symmetry breaking pattern in Eq. (4), which consists of $\textrm{SU}(3)_W$ anti-fundamentals of $\Phi _{ \mathbf {\overline{3}},\rho } \equiv ( \textbf{1}, \mathbf {\overline{3}} , - \frac{1}{3})_{ {\textbf{H}}}^\rho \subset \mathbf {\overline{6}}_{ {\textbf{H}} }^\rho$ (with $\rho =1,2$) and $\Phi _{ \mathbf {\overline{3}} }^\prime \equiv ( \textbf{1}, \mathbf {\overline{3}} , +\frac{2}{3})_{ {\textbf{H}}} \subset \mathbf {15_H}$ after the $\textrm{SU}(6)$ GUT symmetry breaking.

3.1 The Higgs potential

The most general $\textrm{SU}(6)$ Higgs potential contains Higgs fields of $(\mathbf {\overline{6_H}}^\rho , \mathbf {15_H}, \mathbf {35_H} )$. The adjoint Higgs field of $\mathbf {35_H}$ is responsible for the GUT symmetry breaking of $\textrm{SU}(6)\rightarrow {\mathcal {G}}_{331}$. For our purpose, only the Higgs fields of $(\mathbf {\overline{6_H}}^\rho , \mathbf {15_H})$ will be relevant for the sequential symmetry breakings. At the GUT scale, the following terms can be expected in the Higgs potential

$$\begin{aligned} V{} & {} \supset m_{11}^2 | \mathbf {\overline{6_H}}^1 |^2 + m_{22}^2 |\mathbf {\overline{6_H}}^2 |^2 - \left( m_{12}^2 \mathbf {\overline{6_H}}^1 ( \mathbf {\overline{6_H}}^2 )^\dag + H.c. \right) \nonumber \\{} & {} \quad + \frac{ \uplambda _1}{2} | \mathbf {\overline{6_H}}^1 |^4 + \frac{ \uplambda _2}{2} | \mathbf {\overline{6_H}}^2 |^4 \nonumber \\{} & {} \quad + \uplambda _3 | \mathbf {\overline{6_H}}^1 |^2 | \mathbf {\overline{6_H}}^2 |^2 + \uplambda _4 ( \mathbf {\overline{6_H}}^{1\, \dag } \mathbf {\overline{6_H}}^2 ) ( \mathbf {\overline{6_H}}^{2\, \dag } \mathbf {\overline{6_H}}^1 ) \nonumber \\{} & {} \quad + \frac{\uplambda _5 }{2 } \left[ ( \mathbf {\overline{6_H}}^{1\, \dag } \mathbf {\overline{6_H}}^2 )^2 + H.c. \right] \nonumber \\{} & {} \quad + m^2 | \mathbf {15_H} |^2 + \uplambda | \mathbf {15_H} |^4 \nonumber \\{} & {} \quad + \left( \kappa _1 | \mathbf {\overline{6_H}}^1 |^2 + \kappa _2 | \mathbf {\overline{6_H}}^2 |^2 \right) | \mathbf {15_H} |^2 + \kappa _3 (\mathbf {\overline{6_H}}^{1\, \dag } \mathbf {15_H} ) \nonumber \\{} & {} \quad \times (\mathbf {15_H}^\dag \mathbf {\overline{6_H}}^{1} ) + \kappa _4 (\mathbf {\overline{6_H}}^{2\, \dag } \mathbf {15_H} ) (\mathbf {15_H}^\dag \mathbf {\overline{6_H}}^{2} ) \nonumber \\{} & {} \quad + \left( \nu \, \mathbf {\overline{6_H}}^{[ 1} \mathbf {\overline{6_H}}^{2] } \mathbf {15_H} + H.c. \right) . \end{aligned}$$

(18)

The Higgs potential contains the mass squared parameters of $( m_{11}^2 , m_{22}^2 , m_{12}^2, m^2 )$, dimension-one parameter of $\nu$, and dimensionless self couplings of $(\uplambda _{1,\ldots ,5},\uplambda , \kappa _{1,\ldots ,4})$. After the GUT symmetry breaking, we assume all $\textrm{SU}(3)_c$ colored components of $(\mathbf {\overline{6_H}}^\rho , \mathbf {15_H})$ obtain heavy masses of $\Lambda _{\textrm{GUT}}$. The residual massless Higgs fields transforming under the $\textrm{SU}(3)_W \otimes \textrm{U}(1)_X$ symmetry form the following Higgs potential

$$\begin{aligned} V_{\textrm{tot}}= & {} V( \Phi _{ \mathbf {\overline{3}}, \rho } ) {+} V( \Phi _{ \mathbf {\overline{3}}}^\prime ) {+} V( \Phi _{ \mathbf {\overline{3}}, \rho },\Phi _{ \mathbf {\overline{3}}}^\prime ),\end{aligned}$$

(19a)

$$\begin{aligned} V( \Phi _{ \mathbf {\overline{3}}, \rho } )= & {} m_{11}^2 | \Phi _{ \mathbf {\overline{3}}, 1 } |^2 {+} m_{22}^2 | \Phi _{ \mathbf {\overline{3}}, 2 } |^2 \nonumber \\{} & {} {-} \left( m_{12}^2 \Phi _{ \mathbf {\overline{3}}, 1 }^\dag \Phi _{ \mathbf {\overline{3}}, 2 } {+} H.c. \right) \nonumber \\{} & {} {+} \frac{ \uplambda _1}{2} | \Phi _{ \mathbf {\overline{3}}, 1 } |^4 {+} \frac{ \uplambda _2}{2} | \Phi _{ \mathbf {\overline{3}}, 2 } |^4 {+} \uplambda _3 | \Phi _{ \mathbf {\overline{3}}, 1}|^2 |\Phi _{ \mathbf {\overline{3}}, 2 }|^2 \nonumber \\{} & {} {+} \uplambda _4 (\Phi _{ \mathbf {\overline{3}}, 1 }^\dag \Phi _{ \mathbf {\overline{3}}, 2 } ) (\Phi _{ \mathbf {\overline{3}}, 2 }^\dag \Phi _{ \mathbf {\overline{3}}, 1 } ) \nonumber \\{} & {} {+} \frac{\uplambda _5}{2} \left[ (\Phi _{ \mathbf {\overline{3}}, 1 }^\dag \Phi _{ \mathbf {\overline{3}}, 2 } )^2 {+} H.c. \right] , \end{aligned}$$

(19b)

$$\begin{aligned} V( \Phi _{ \mathbf {\overline{3}}}^\prime )= & {} m^2 | \Phi _{ \mathbf {\overline{3}}}^\prime |^2 {+} \uplambda | \Phi _{ \mathbf {\overline{3}}}^\prime |^4, \end{aligned}$$

(19c)

$$\begin{aligned} V( \Phi _{ \mathbf {\overline{3}}, \rho },\Phi _{ \mathbf {\overline{3}}}^\prime )= & {} \left( \kappa _1 |\Phi _{ \mathbf {\overline{3}},1} |^2 {+} \kappa _2 |\Phi _{ \mathbf {\overline{3}},2} |^2 \right) | \Phi _{ \mathbf {\overline{3}}}^\prime |^2 \nonumber \\{} & {} {+} \kappa _3 (\Phi _{ \mathbf {\overline{3}}, 1 }^\dag \Phi _{ \mathbf {\overline{3}}}^\prime ) (\Phi _{ \mathbf {\overline{3}}}^{\prime \,\dag } \Phi _{ \mathbf {\overline{3}}, 1 } ) \nonumber \\{} & {} {+} \kappa _4 (\Phi _{ \mathbf {\overline{3}}, 2 }^\dag \Phi _{ \mathbf {\overline{3}}}^\prime ) (\Phi _{ \mathbf {\overline{3}}}^{\prime \,\dag } \Phi _{ \mathbf {\overline{3}}, 2 } ) \nonumber \\{} & {} {+} \left( \nu \epsilon _{IJK} (\Phi _{ \mathbf {\overline{3}},1})_I ( \Phi _{ \mathbf {\overline{3}}, 2} )_J (\Phi _{ \mathbf {\overline{3}}}^\prime )_K {+} H.c. \right) .\nonumber \\ \end{aligned}$$

(19d)

The last $\nu$-term in Eq. (19d) is inevitable by both the gauge symmetry and the emergent global $\textrm{SU}(2)_F$ symmetry in Eq. (2), with $I,J,K=1,2,3$ being the $\textrm{SU}(3)_W$ anti-fundamental indices.

We denote the Higgs fields under the $\textrm{SU}(3)_W \otimes \textrm{U}(1)_{X}$ representations as follows

$$\begin{aligned}{} & {} \Phi _{ \mathbf {\overline{3}}, \rho } = \frac{1}{ \sqrt{2}} \left( \begin{array}{c} \sqrt{2} \phi _\rho ^- \\ \phi _\rho - i \eta _\rho \\ h_\rho - i\pi _\rho \end{array} \right) ,\quad \Phi _{ \mathbf {\overline{3}} }^\prime = \frac{1}{ \sqrt{2}} \left( \begin{array}{c} h_u + i \pi _u \\ \sqrt{2} \chi ^+ \\ \sqrt{2} \chi ^{\prime \,+} \end{array} \right) ,\nonumber \\ \end{aligned}$$

(20)

where the electric charges are given according to Eq. (6). According to the VEV assignment in Eq. (12), we expect the non-vanishing Higgs VEVs of

$$\begin{aligned}{} & {} \langle h_1 \rangle = V_1 = V_{331} c_{{\tilde{\beta }}},\quad \langle h_2 \rangle = V_2 = V_{331} s_{{\tilde{\beta }}},\quad \langle h_u \rangle = v_u,\nonumber \\ \end{aligned}$$

(21)

with

$$\begin{aligned} t_{{\tilde{\beta }} }\equiv & {} \frac{V_2}{ V_1} \end{aligned}$$

(22)

parametrizing the ratio between two 331 symmetry breaking VEVs. Accordingly, the minimization of the Higgs potential in Eqs. (19) leads to the following conditions

$$\begin{aligned} \frac{ \partial V}{ \partial V_1} =0\Rightarrow & {} m_{11}^2= m_{12}^2 t_{{\tilde{\beta }}} - \frac{ \uplambda _1}{2} V_1^2 \nonumber \\{} & {} - \frac{ \uplambda _{3} + \uplambda _4 + \uplambda _5 }{2} V_2^2 - \frac{ \kappa _{1}}{2} v_u^2,\end{aligned}$$

(23a)

$$\begin{aligned} \frac{ \partial V}{ \partial V_2} =0\Rightarrow & {} m_{22}^2= \frac{ m_{12}^2 }{ t_{{\tilde{\beta }}} } - \frac{ \uplambda _2}{2} V_2^2\nonumber \\{} & {} - \frac{ \uplambda _{3} + \uplambda _4 + \uplambda _5 }{2} V_1^2 - \frac{ \kappa _{2}}{2} v_u^2,\end{aligned}$$

(23b)

$$\begin{aligned} \frac{ \partial V}{ \partial v_u} =0\Rightarrow & {} m^2= -\uplambda v_u^2 - \frac{1}{2}( \kappa _{1} V_1^2 + \kappa _{2} V_2^2 ). \end{aligned}$$

(23c)

Note that the $\nu$-term that mixes the $\Phi _{ \mathbf {\overline{3}},\rho }$ and the $\Phi _{ \mathbf {\overline{3}} }^\prime$ does not enter the minimization condition with the VEV assignment in Eq. (12), while this term will be important in generating the fermion masses. Correspondingly, this leads to an unwanted tadpole term of $- \frac{1}{2 \sqrt{2} } \nu ( \phi _1 V_2 - \phi _2 V_1) v_u$ in the Higgs potential.

To resolve the tadpole problem, the only way is to develop EWSB VEVs of

$$\begin{aligned}{} & {} \langle \phi _1\rangle = u_1=v_\phi c_{\beta ^\prime },\quad \langle \phi _2\rangle = u_2 = v_\phi s_{\beta ^\prime }, \end{aligned}$$

(24)

presumably with $v_\phi \sim {\mathcal {O}}(v_{\textrm{EW}} )$ as was considered in Refs. [40,41,42,43]. The Nambu–Goldstone boson (NGB) of $\xi ^0$ can be obtained from the following derivative terms

$$\begin{aligned}{} & {} \sim \frac{i }{ 2\sqrt{2} } g_{3L} ( N_\mu - {\bar{N}}_\mu ) \partial ^\mu \nonumber \\{} & {} \quad \times \left[ ( u_1 h_1 + u_2 h_2 ) - ( V_1 \phi _1 + V_2 \phi _2 ) \right] \nonumber \\{} & {} \quad \Rightarrow \xi ^0= c_\theta ( c_{{\tilde{\beta }}} \phi _1 + s_{{\tilde{\beta }}} \phi _2 ) - s_\theta ( c_{\beta ^\prime } h_1 + s_{\beta ^\prime } h_2 ), \end{aligned}$$

(25)

with

$$\begin{aligned} t_\theta\equiv & {} \frac{ v_\phi }{ V_{331} } \end{aligned}$$

(26)

parametrizing the ratio between two symmetry-breaking scales in each $\textrm{SU}(3)_W$ anti-fundamental Higgs of $\Phi _{ \mathbf {\overline{3}},\rho }$. With a natural assumption of the following VEV orthogonal relation of

$$\begin{aligned}{} & {} \sum _\rho u_\rho V_\rho =0 \Rightarrow \beta ^\prime = {\tilde{\beta }} - \frac{\pi }{2}, \end{aligned}$$

(27)

the potential mixing between the $(W_\mu ^\pm , C_\mu ^\pm )$, as well as the $( N_\mu , {\bar{N}}_\mu , Z_\mu ^\prime )$, can be avoided. This can be confirmed with the explicit gauge fields in terms of a $3\times 3$ matrix given in Eq. (66). Thus, the VEVs in Eq. (24) become

$$\begin{aligned}{} & {} \langle \phi _1\rangle = u_1=v_\phi s_{{\tilde{\beta }} },\quad \langle \phi _2\rangle = u_2 = - v_\phi c_{{\tilde{\beta }} }. \end{aligned}$$

(28)

The minimization conditions of the Higgs potential in Eqs. (23) are modified to be

$$\begin{aligned} \frac{\partial V}{\partial V_1} =0\Rightarrow & {} m_{11}^2 = m_{12}^2 t_{{\tilde{\beta }}} - \frac{\uplambda _1}{2}( V_1^2 + u_1^2 ) - \frac{ \uplambda _{3}}{2} ( V_2^2 + u_2^2 ) \nonumber \\{} & {} - \frac{ \uplambda _4 + \uplambda _5 }{2} ( u_1 u_2 + V_1 V_2 ) t_{{\tilde{\beta }}} - \frac{ \kappa _{1}}{2} v_u^2 \nonumber \\{} & {} + \frac{ \nu u_2 v_u}{ \sqrt{2} V_1}, \end{aligned}$$

(29a)

$$\begin{aligned} \frac{\partial V}{\partial V_2} =0\Rightarrow & {} m_{22}^2 = \frac{ m_{12}^2}{ t_{{\tilde{\beta }}} } - \frac{ \uplambda _2}{2}( V_2^2 + u_2^2 ) -\frac{ \uplambda _{3}}{2} ( V_1^2 + u_1^2 ) \nonumber \\{} & {} - \frac{ \uplambda _4 + \uplambda _5 }{2} ( u_1 u_2 + V_1 V_2 ) \frac{1 }{t_{{\tilde{\beta }}}} - \frac{ \kappa _{2}}{2} v_u^2\nonumber \\{} & {} - \frac{ \nu u_1 v_u}{ \sqrt{2} V_2}, \end{aligned}$$

(29b)

$$\begin{aligned} \frac{\partial V}{\partial v_u} =0\Rightarrow & {} - m^2 = \uplambda v_u^2 + \frac{\kappa _{1} }{2}(V_1^2 + u_1^2 ) + \frac{\kappa _{2}}{2} ( V_2^2 + u_2^2 ) \nonumber \\{} & {} + \frac{ \nu }{ \sqrt{2} v_u} ( u_1 V_2 - u_2 V_1 ), \end{aligned}$$

(29c)

$$\begin{aligned} \frac{\partial V}{\partial u_1} =0\Rightarrow & {} m_{11}^2 = - \frac{m_{12}^2}{ t_{{\tilde{\beta }}} } - \frac{\uplambda _1 }{2} ( V_1^2 + u_1^2 ) - \frac{ \uplambda _{3}}{2} ( V_2^2 + u_2^2 ) \nonumber \\{} & {} + \frac{ \uplambda _4 + \uplambda _5 }{2} ( u_1 u_2 + V_1 V_2 ) \frac{1 }{t_{{\tilde{\beta }}}} - \frac{ \kappa _{1}}{2} v_u^2\nonumber \\{} & {} - \frac{ \nu V_2 v_u}{ \sqrt{2} u_1}, \end{aligned}$$

(29d)

$$\begin{aligned} \frac{\partial V}{\partial u_2} = 0\Rightarrow & {} m_{22}^2 = {-}m_{12}^2 t_{ {\tilde{\beta }} } {-} \frac{ \uplambda _2}{2}( V_2^2 {+} u_2^2 ) {-}\frac{ \uplambda _{3}}{2} ( V_1^2 {+} u_1^2 ) \nonumber \\{} & {} {+} \frac{ \uplambda _4 {+} \uplambda _5 }{2} ( u_1 u_2 {+} V_1 V_2 ) t_{{\tilde{\beta }} } {-} \frac{ \kappa _{2}}{2} v_u^2\nonumber \\{} & {} {+} \frac{ \nu V_1 v_u}{ \sqrt{2} u_2}. \end{aligned}$$

(29e)

By equating Eqs. (29a) with (29d), and Eqs. (29b) with (29e), we have a constraint of

$$\begin{aligned}{} & {} t_\theta ^3 -\frac{ \sqrt{2} \nu v_{u} }{(\uplambda _{4}+\uplambda _{5}) V_{331}^2 } t_{\theta }^2 + \left( \frac{ 2 m_{12}^2 }{s_{ {\tilde{\beta }}} c_{ {\tilde{\beta }}} (\uplambda _{4}+\uplambda _{5}) V_{331}^2 } - 1 \right) t_\theta \nonumber \\{} & {} \quad + \frac{ \sqrt{2} \nu v_{u} }{(\uplambda _{4}+\uplambda _{5}) V_{331}^2 }=0, \end{aligned}$$

(30)

with the relation in Eq. (27). Consider the scale hierarchy of $m_{12}^2 \sim {\mathcal {O}}(V_{331}^2)$, $\nu \sim v_u \sim {\mathcal {O}}(v_{\textrm{EW}})$. It is straightforward to expect $t_\theta \sim {\mathcal {O}}( (v_{\textrm{EW}}/ V_{331} )^2 )$ from Eq. (30). Thus, a natural scale of $v_\phi$ can be further suppressed from the EW scale of $v_{\textrm{EW}}$, such as $\sim {\mathcal {O}}(1)\,\textrm{GeV}$. This was not noted in the previous Refs. [40,41,42,43] with the similar VEV assignment. Here and after, we will consider the following parameter inputs⁹

$$\begin{aligned} \nu\sim & {} {-}{\mathcal {O}}(100)\,\textrm{GeV}, v_u \approx 246\,\textrm{GeV}, v_\phi \sim {\mathcal {O}}(1)\,\textrm{GeV},\nonumber \\ V_{331}\sim & {} {\mathcal {O}}(10)\,\textrm{TeV} \nonumber \\ t_{{\tilde{\beta }} }\sim & {} {\mathcal {O}}(1), \quad m_{12}^2 \sim {\mathcal {O}}(100)\,\textrm{TeV}^2, \end{aligned}$$

(31)

instead of performing the detailed parameter scans. The choice of $V_{331}\sim {\mathcal {O}}(10)\,\textrm{TeV}$ will become clear from the $(b,\tau )$ Yukawa couplings with the $125\,\textrm{GeV}$ SM-like Higgs boson. Notice that in the decoupling limit of $m_{12}\sim V_{331}\rightarrow \infty$, one naturally has $t_\theta \rightarrow 0$ and $v_\phi \rightarrow 0$ from Eq. (30).

3.2 The charged and CP-odd Higgs bosons

The charged Higgs bosons of $\Phi _{\pm }=( \phi _1^\pm , \phi _2^\pm , \chi ^\pm , \chi ^{\prime \, \pm })$ form the mass squared matrix of

$$\begin{aligned} ( {\mathcal {M}}_{\pm }^2)_{ 4\times 4}= & {} \left( \begin{array}{cccc} M_{\phi _1^+ \phi _1^-}^2 &{}\quad M_{\phi _1^+ \phi _2^- }^2 &{}\quad M_{\phi _1^+ \chi ^- }^2 &{}\quad M_{\phi _1^+ \chi ^{\prime \,-} }^2 \\ M_{\phi _1^- \phi _2^+ }^2 &{}\quad M_{\phi _2^+ \phi _2^- }^2 &{}\quad M_{\phi _2^+ \chi ^- }^2 &{}\quad M_{\phi _2^+ \chi ^{\prime \, -} }^2 \\ M_{\phi _1^- \chi ^+ }^2 &{}\quad M_{\phi _2^- \chi ^+ }^2 &{}\quad M_{\chi ^+ \chi ^- }^2 &{}\quad M_{\chi ^+ \chi ^{\prime \,-} }^2 \\ M_{\phi _1^- \chi ^{\prime \,+} }^2 &{}\quad M_{\phi _2^- \chi ^{\prime \, +} }^2 &{}\quad M_{\chi ^- \chi ^{\prime \, +} }^2 &{}\quad M_{\chi ^{\prime \,+} \chi ^{\prime \,-} }^2 \end{array} \right) \nonumber \\{} & {} \sim \left( \begin{array}{cccc} 1 &{}\quad 1 &{}\quad \epsilon &{}\quad \epsilon \\ 1 &{}\quad 1 &{}\quad \epsilon &{}\quad \epsilon \\ \epsilon &{}\quad \epsilon &{}\quad \epsilon ^2 &{}\quad \epsilon ^2 \\ \epsilon &{}\quad \epsilon &{}\quad \epsilon ^2 &{}\quad 1 \\ \end{array} \right) V_{331}^2, \end{aligned}$$

(32)

with a small parameter given by $\epsilon \equiv \frac{v_u}{V_{331} }\sim \frac{v_\phi }{ \nu } \sim {\mathcal {O}}(0.01)$. The orthogonal transformations to gauge eigenstates of $\Phi _{\pm }$ are expressed as follows

$$\begin{aligned} \left( \begin{array}{c} G^\pm \\ G^{\prime \, \pm } \\ H_1^\pm \\ H_2^\pm \end{array} \right)= & {} {\widetilde{V}}_{\pm } \cdot \left( \begin{array}{c} \phi _1^\pm \\ \phi _2^\pm \\ \chi ^\pm \\ \chi ^{\prime \, \pm } \end{array} \right) , \nonumber \\ {\widetilde{V}}_{\pm }= & {} \left( \begin{array}{cccc} c_{\tau _1} &{}\quad 0 &{}\quad 0 &{}\quad -s_{\tau _1} \\ 0 &{}\quad 1 &{}\quad 0 &{}\quad 0 \\ 0 &{}\quad 0 &{}\quad 1 &{}\quad 0 \\ s_{\tau _1} &{}\quad 0 &{}\quad 0 &{}\quad c_{\tau _1} \\ \end{array} \right) \nonumber \\{} & {} \cdot \left( \begin{array}{cccc} 1 &{}\quad 0 &{}\quad 0 &{}\quad 0 \\ 0 &{}\quad c_{\tau _2} &{}\quad -s_{\tau _2} &{}\quad 0 \\ 0 &{}\quad s_{\tau _2} &{}\quad c_{\tau _2} &{}\quad 0 \\ 0 &{}\quad 0 &{}\quad 0 &{}\quad 1 \\ \end{array} \right) \cdot \left( \begin{array}{cccc} c_{{\tilde{\beta }}} &{}\quad s_{{\tilde{\beta }}} &{}\quad 0 &{}\quad 0 \\ - s_{{\tilde{\beta }}} &{}\quad c_{{\tilde{\beta }}} &{}\quad 0 &{}\quad 0 \\ 0 &{}\quad 0 &{}\quad 1 &{}\quad 0 \\ 0 &{}\quad 0 &{}\quad 0 &{}\quad 1 \\ \end{array} \right) \nonumber \\= & {} \left( \begin{array}{cccc} c_{\tau _1} c_{{\tilde{\beta }}} &{}\quad c_{\tau _1} s_{{\tilde{\beta }}} &{}\quad 0 &{}\quad -s_{\tau _1} \\ -c_{\tau _2} s_{{\tilde{\beta }}} &{}\quad c_{\tau _2} c_{{\tilde{\beta }}} &{}\quad -s_{\tau _2} &{}\quad 0 \\ -s_{\tau _2} s_{{\tilde{\beta }}} &{}\quad s_{\tau _2} c_{{\tilde{\beta }}} &{}\quad c_{\tau _2} &{}\quad 0 \\ s_{\tau _1} c_{{\tilde{\beta }}} &{}\quad s_{\tau _1} s_{{\tilde{\beta }}} &{}\quad 0 &{}\quad c_{\tau _1} \\ \end{array} \right) , \end{aligned}$$

(33)

with $t_{\tau _1} \equiv \frac{v_u}{ V_{331}}$ and $t_{\tau _2} \equiv \frac{v_\phi }{v_u}$. Two corresponding non-zero eigenvalues for two charged Higgs bosons of $H_{1,2}^\pm$ are given by

$$\begin{aligned} M_{H_1^\pm }^2= & {} \frac{{\mathcal {B}}}{ 16 V_{331} v_u v_\phi } \left( 1 {-} \sqrt{1 {+} \frac{{\mathcal {A}}}{ {\mathcal {B}}^2}} \right) , \nonumber \\ M_{H_2^\pm }^2= & {} \frac{ {\mathcal {B}}}{ 16 V_{331} v_u v_\phi } \left( 1 {+} \sqrt{1 {+} \frac{{\mathcal {A}}}{{\mathcal {B}}^2}} \right) , \nonumber \\ {\mathcal {A}}= & {} 32 \sqrt{2} \nu v_{u} v_{\phi } (v_{u}^2{+}v_{\phi }^2)(\kappa _{3}+\kappa _{4}+(\kappa _{3}-\kappa _{4}) c_{2 {\tilde{\beta }}}) V_{331}^5 \nonumber \\{} & {} - 64 v_{\phi }^2(v_{u}^2+v_{\phi }^2)(2\nu ^2+\kappa _{3} \kappa _{4} v_{u}^2) V_{331}^4 \nonumber \\{} & {} + 32 \sqrt{2} \nu v_{u} v_{\phi } (v_{u}^2+v_{\phi }^2)((v_{u}^2+v_{\phi }^2)(\kappa _{3}+\kappa _{4})\nonumber \\{} & {} +(v_{u}^2-v_{\phi }^2)(\kappa _{3}-\kappa _{4}) c_{2 {\tilde{\beta }}}) V_{331}^3 \nonumber \\{} & {} -64 v_{u}^2 v_{\phi }^2 (v_{u}^2+v_{\phi }^2)(2 \nu ^2+\kappa _{3} \kappa _{4} v_{u}^2) V_{331}^2\nonumber \\{} & {} - 32 \sqrt{2} \nu v_{u}^3 v_{\phi }^3 (v_{u}^2 + v_{\phi }^2)((\kappa _{3}-\kappa _{4})c_{2 {\tilde{\beta }}}-\kappa _{3}-\kappa _{4}) \nonumber \\{} & {} V_{331}\sim \nu v_\phi v_u^3 V_{331}^5, \nonumber \\ {\mathcal {B}}= & {} 2 V_{331} v_{u} v_{\phi } \left[ c_{2 {\tilde{\beta }}} (\kappa _{3}-\kappa _{4}) (V_{331}^2-v_{\phi }^2) \right. \nonumber \\{} & {} \left. + (\kappa _{3}+\kappa _{4}) (V_{331}^2+v_{\phi }^2) \right] \nonumber \\{} & {} -8 \sqrt{2} \nu V_{331}^2 v_{\phi }^2 - 4 \sqrt{2} \nu v_{u}^2 (V_{331}^2+v_{\phi }^2)\nonumber \\{} & {} +4 V_{331} v_{u}^3 v_{\phi } (\kappa _{3}+\kappa _{4}) \nonumber \\\sim & {} v_u v_\phi V_{331}^3. \end{aligned}$$

(34)

A simple expansion of Eq. (34) in terms of the mass hierarchy assumed in Eq. (31) leads to the approximated mass scales of

$$\begin{aligned}{} & {} M_{H_1^\pm }^2 \sim {\mathcal {O}}( V_{331}^2 ),\quad M_{H_2^\pm }^2 \sim {\mathcal {O}}( V_{331}^2 ). \end{aligned}$$

(35)

The CP-odd Higgs bosons form the mass squared matrix of

$$\begin{aligned} ( {\mathcal {M}}_{0^- }^2)_{ 5\times 5}= & {} \left( \begin{array}{ccccc} M_{\pi _u \pi _u}^2 &{}\quad M_{\pi _u \eta _1}^2 &{}\quad M_{\pi _u \eta _2}^2 &{}\quad M_{\pi _u \pi _1}^2 &{}\quad M_{\pi _u \pi _2}^2 \\ M_{\pi _u \eta _1}^2 &{}\quad M_{\eta _1 \eta _1}^2 &{}\quad M_{\eta _1 \eta _2}^2 &{}\quad M_{\pi _1 \eta _1}^2 &{}\quad M_{\pi _2 \eta _1}^2 \\ M_{\pi _u \eta _2}^2 &{}\quad M_{\eta _1 \eta _2}^2 &{}\quad M_{\eta _2 \eta _2}^2 &{}\quad M_{\pi _1 \eta _2}^2 &{}\quad M_{\pi _2 \eta _2}^2 \\ M_{\pi _u \pi _1}^2 &{}\quad M_{\pi _1 \eta _1 }^2 &{}\quad M_{\pi _1 \eta _2}^2 &{}\quad M_{\pi _1 \pi _1}^2 &{}\quad M_{\pi _1 \pi _2}^2 \\ M_{\pi _u \pi _2}^2 &{}\quad M_{\pi _2 \eta _1}^2 &{}\quad M_{\pi _2 \eta _2}^2 &{}\quad M_{\pi _1 \pi _2}^2 &{}\quad M_{\pi _2 \pi _2}^2 \\ \end{array} \right) \nonumber \\\sim & {} \left( \begin{array}{ccccc} \epsilon ^2 &{}\quad \epsilon &{}\quad \epsilon &{}\quad \epsilon ^3 &{}\quad \epsilon ^3 \\ \epsilon &{}\quad 1 &{}\quad 1 &{}\quad \epsilon ^2 &{}\quad \epsilon ^2 \\ \epsilon &{}\quad 1 &{}\quad 1 &{}\quad \epsilon ^2 &{}\quad \epsilon ^2 \\ \epsilon ^3 &{}\quad \epsilon ^2 &{}\quad \epsilon ^2 &{}\quad 1 &{}\quad 1 \\ \epsilon ^3 &{}\quad \epsilon ^2 &{}\quad \epsilon ^2 &{}\quad 1 &{}\quad 1 \\ \end{array} \right) V_{331}^2, \end{aligned}$$

(36)

in the basis of $\Phi _{0^-}=(\pi _u, \eta _{1,2}, \pi _{1,2})$. We find three zero eigenvalues corresponding to three massless NGBs with the constraint in Eq. (30). The orthogonal transformations to $\Phi _{0^-}$ are expressed as follows

$$\begin{aligned} \left( \begin{array}{c} G^0 \\ G^{0\,\prime } \\ G^{0\,\prime \prime } \\ A^{0} \\ A^{0\,\prime } \end{array} \right)= & {} {\widetilde{V}}_{0^-} \cdot \left( \begin{array}{c} \pi _u \\ \eta _1 \\ \eta _2 \\ \pi _1 \\ \pi _2 \\ \end{array} \right) ,\nonumber \\ {\widetilde{V}}_{0^-}= & {} \left( \begin{array}{ccccc} 0 &{}\quad c_\theta c_{{\tilde{\beta }}} &{}\quad c_\theta s_{{\tilde{\beta }}} &{}\quad s_\theta s_{{\tilde{\beta }}} &{}\quad -s_\theta c_{{\tilde{\beta }}} \\ 0 &{}\quad - s_\theta s_{{\tilde{\beta }}} &{}\quad s_\theta c_{{\tilde{\beta }}} &{}\quad c_\theta c_{{\tilde{\beta }}} &{}\quad c_\theta s_{{\tilde{\beta }}} \\ c_\tau &{}\quad s_\tau c_\theta s_{{\tilde{\beta }}} &{}\quad - s_\tau c_\theta c_{{\tilde{\beta }}} &{}\quad s_\tau s_\theta c_{{\tilde{\beta }}} &{}\quad s_\tau s_\theta s_{{\tilde{\beta }}} \\ -s_\tau &{}\quad c_\tau c_\theta s_{{\tilde{\beta }}} &{}\quad -c_\tau c_\theta c_{{\tilde{\beta }}} &{}\quad c_\tau s_\theta c_{{\tilde{\beta }}} &{}\quad c_\tau s_\theta s_{{\tilde{\beta }}} \\ 0 &{}\quad s_\theta c_{{\tilde{\beta }}} &{}\quad s_\theta s_{{\tilde{\beta }}} &{}\quad - c_\theta s_{{\tilde{\beta }}} &{}\quad c_\theta c_{{\tilde{\beta }}} \\ \end{array} \right) , \end{aligned}$$

(37)

with $t_\tau \equiv \frac{ v_\phi V_{331} }{ \sqrt{ V_{331}^2 + v_\phi ^2 } v_u } \approx \frac{v_\phi }{v_u }$. Two corresponding non-zero eigenvalues for two CP-odd Higgs bosons are given by

$$\begin{aligned} M_{A^0 }^2= & {} \frac{-\sqrt{2} \nu (V_{331}^2 + v_\phi ^2 ) v_u^2+(\uplambda _{4} - \uplambda _5 ) V_{331} v_\phi ( V_{331}^2 + v_{\phi }^2 ) v_u }{2 V_{331} v_{\phi } v_u} \nonumber \\\sim & {} {\mathcal {O}}( V_{331}^2 ),\nonumber \\ M_{A^{0\,\prime } }^2= & {} - \nu \frac{ V_{331}^2 ( v_u^2 + v_\phi ^2 ) + v_{\phi }^2 v_u^2}{ \sqrt{2} V_{331} v_{\phi } v_u} \sim {\mathcal {O}}( V_{331}^2 ). \end{aligned}$$

(38)

Hence, we do not expect the discovery of these two CP-odd Higgs bosons at the current LHC direct searches.

3.3 The CP-even Higgs bosons

There are five CP-even Higgs fields of $(h_u, \phi _{1,2}, h_{1,2})$ in the gauge eigenbasis, and one of their linear combination will be identified as the NGB. Their masses and mixings play the key role in generating the bottom quark and tau lepton masses, as well as determining their Yukawa couplings with the SM-like Higgs boson. It takes two steps to obtain their mass eigenstates. To see that, we first perform the orthonormal transformations to $( \phi _{1,2}, h_{1,2} )$ as follows

$$\begin{aligned} \left( \begin{array}{c} \xi ^0 \\ \phi ^0 \\ h_1^\prime \\ h_2^\prime \\ \end{array} \right)= & {} {\widetilde{V}}_{0^+} \cdot \left( \begin{array}{c} \phi _1 \\ \phi _2 \\ h_1 \\ h_2 \\ \end{array} \right) , \quad {\widetilde{V}}_{0^+}= \small { \left( \begin{array}{cccc} c_\theta c_{{\tilde{\beta }}} &{}\quad c_\theta s_{{\tilde{\beta }}} &{}\quad - s_\theta s_{{\tilde{\beta }}} &{}\quad s_\theta c_{{\tilde{\beta }}} \\ - c_\theta s_{{\tilde{\beta }}} &{}\quad c_\theta c_{{\tilde{\beta }}} &{}\quad - s_\theta c_{{\tilde{\beta }}} &{}\quad - s_\theta s_{{\tilde{\beta }}} \\ s_\theta c_{{\tilde{\beta }}} &{}\quad s_\theta s_{{\tilde{\beta }}} &{}\quad c_\theta s_{{\tilde{\beta }}} &{}\quad - c_\theta c_{{\tilde{\beta }}} \\ - s_\theta s_{{\tilde{\beta }}} &{}\quad s_\theta c_{{\tilde{\beta }}} &{}\quad c_\theta c_{{\tilde{\beta }}} &{}\quad c_\theta s_{{\tilde{\beta }}} \\ \end{array} \right) },\nonumber \\ \end{aligned}$$

(39)

with $\xi ^0$ being the massless NGB. Under the basis of $(h_u, \phi ^0, h_{1,2}^\prime )$, the remaining four CP-even Higgs fields form the mass squared matrix and can be expanded as follows

$$\begin{aligned} ( {\mathcal {M}}_{0^+ }^2)_{ 4\times 4}= & {} \left( \begin{array}{cccc} M_{h_u h_u}^2 &{}\quad M_{ h_u \phi ^0 }^2 &{}\quad M_{h_u h_1^\prime }^2 &{}\quad M_{h_u h_2^\prime }^2 \\ M_{h_u \phi ^0 }^2 &{}\quad M_{\phi ^0 \phi ^0 }^2 &{}\quad M_{\phi ^0 h_1^\prime }^2 &{}\quad M_{\phi ^0 h_2^\prime }^2 \\ M_{h_u h_1^\prime }^2 &{}\quad M_{\phi ^0 h_1^\prime }^2 &{}\quad M_{ h_1^\prime h_1^\prime }^2 &{}\quad M_{ h_1^\prime h_2^\prime }^2 \\ M_{h_u h_2^\prime }^2 &{}\quad M_{\phi ^0 h_2^\prime }^2 &{}\quad M_{ h_1^\prime h_2^\prime }^2 &{}\quad M_{ h_2^\prime h_2^\prime }^2 \\ \end{array} \right) \nonumber \\\sim & {} \left( \begin{array}{cccc} \epsilon ^2 &{}\quad \epsilon &{}\quad \epsilon &{}\quad \epsilon \\ \epsilon &{}\quad 1 &{}\quad \epsilon ^2 &{}\quad \epsilon ^2 \\ \epsilon &{}\quad \epsilon ^2 &{}\quad 1 &{}\quad 1 \\ \epsilon &{}\quad \epsilon ^2 &{}\quad 1 &{}\quad 1 \\ \end{array} \right) V_{331}^2 \nonumber \\= & {} ( {\mathcal {M}}_{0^+ }^2)^{(0)} + ( {\mathcal {M}}_{0^+ }^2)^{(1)} + ( {\mathcal {M}}_{0^+ }^2)^{(2)}, \end{aligned}$$

(40)

with a small parameter given by $\epsilon \equiv \frac{v_u}{V_{331} }\sim {\mathcal {O}}(0.01)$. The further diagonalization of Eq. (40) transforms into mass eigenstates of $(H_u, H_\phi , H_1, H_2)$ such that

$$\begin{aligned}{} & {} \left( \begin{array}{c} H_u \\ H_\phi \\ H_1 \\ H_2 \end{array} \right) = V_{0^+}\cdot \left( \begin{array}{c} h_u \\ \phi ^0 \\ h_1^\prime \\ h_2^\prime \\ \end{array} \right) , \nonumber \\{} & {} \quad V_{0^+} ( {\mathcal {M}}_{0^+ }^2) V_{0^+}^T = \textrm{diag} ( M_{H_u}^2, M_{H_\phi }^2, M_{H_1}^2, M_{H_2}^2 ). \end{aligned}$$

(41)

Among them, $H_u$ is the lightest CP-even Higgs boson with mass of 125 GeV, while others have masses of $\sim {\mathcal {O}}(V_{331})$.

To have positive definite eigenvalues for CP-even Higgs boson mass squares in Eq. (40), one cannot have the $\nu$ parameter as large as $V_{331}$. That is why we chose $\nu \sim {\mathcal {O}}(100)\,\textrm{GeV}$ in Eq. (31). However, a $\nu$-problem emerges, namely, why is a mass parameter from a 331-invariant Higgs potential takes a value comparable to the EW scale. This problem is analogous to the well-known $\mu$-problem in the minimal supersymmetric Standard Model (MSSM) [68]. One can thus expect this $\nu$-term to originate from some non-renormalizable terms in the realistic non-minimal GUTs with $n_g=3$.¹⁰ This type of terms are inevitable due to the gravitational effects that break the global $\textrm{U}(1)$ symmetry explicitly.¹¹ Taking the $\textrm{SU}(8)$ GUT as an example again, one such possible $d=5$ non-renormalizable term is expected to be

$$\begin{aligned}{} & {} \textrm{SU}(8)~:~ \frac{1}{ M_{\textrm{pl}} } \epsilon _{\rho _1\ldots \rho _4 } \mathbf {\overline{8_H}}^{\rho _1}\ldots \mathbf {\overline{8_H}}^{\rho _4 } \cdot \mathbf {70_H} \nonumber \\{} & {} \quad \supset \frac{1}{ M_{\textrm{pl}} } \epsilon _{\rho _1\ldots \rho _4 } \left( \mathbf {\overline{4}}, \textbf{1}, +\frac{1}{4} \right) _{ {\textbf{H}}}^{\rho _1} \otimes \left( \textbf{1}, \mathbf {\overline{4}}, -\frac{1}{4} \right) _{ {\textbf{H}}}^{\rho _2} \nonumber \\{} & {} \quad \otimes \left( \textbf{1}, \mathbf {\overline{4}}, -\frac{1}{4} \right) _{ {\textbf{H}}}^{\rho _3} \otimes \left( \textbf{1}, \mathbf {\overline{4}}, -\frac{1}{4} \right) _{ {\textbf{H}}}^{\rho _4}\otimes \left( \textbf{4}, \mathbf {\overline{4}}, +\frac{1}{2} \right) _{ {\textbf{H}}} \nonumber \\{} & {} \quad \supset \frac{ V_{441} }{ M_{\textrm{pl}} } \epsilon _{\rho _2\ldots \rho _4 } \, \left( \textbf{1}, \mathbf {\overline{4}}, -\frac{1}{4} \right) _{ {\textbf{H}}}^{\rho _2} \otimes \left( \textbf{1}, \mathbf {\overline{4}}, -\frac{1}{4} \right) _{ {\textbf{H}}}^{\rho _3} \nonumber \\{} & {} \quad \otimes \left( \textbf{1}, \mathbf {\overline{4}}, -\frac{1}{4}\right) _{ {\textbf{H}}}^{\rho _4} \otimes \left( \textbf{1}, \mathbf {\overline{4}}, +\frac{3}{4} \right) _{ {\textbf{H}}} \sim \frac{ V_{441} V_{341} }{ M_{\textrm{pl}} } \epsilon _{ \rho _3 \rho _4} \nonumber \\{} & {} \quad \, \left( \textbf{1}, \mathbf {\overline{3}}, -\frac{1}{3} \right) _{ {\textbf{H}}}^{\rho _3} \otimes \left( \textbf{1}, \mathbf {\overline{3}}, -\frac{1}{3} \right) _{ {\textbf{H}}}^{\rho _4} \otimes \left( \textbf{1}, \mathbf {\overline{3}}, +\frac{2}{3}\right) _{ {\textbf{H}}}^{\prime \prime \prime }. \end{aligned}$$

(42)

Here, the decompositions are achieved according to Eqs. (17). Obviously, this non-renormalizable term induced by the gravitational effect reproduces what we considered as the $\nu$-term in Eq. (19d). Thus, the value of $\nu \sim {\mathcal {O}}(100)\,\textrm{GeV}$ in Eq. (31) can be naturally realized with $V_{441}\sim {\mathcal {O}}(10^{12})\,\textrm{GeV}$ and $V_{341}\sim {\mathcal {O}}(10^{9})\,\textrm{GeV}$.

With the hierarchies of mass parameter in Eq. (31), the diagonalization of mass matrix in Eq. (40) can be achieved in terms of perturbation. Hence, we express the mixing matrix in Eq. (41) as

$$\begin{aligned} V_{0^+}= & {} {\widetilde{U}} {\widetilde{U}}^{(0) }. \end{aligned}$$

(43)

At the leading order, it is straightforward to diagonalize the $( {\mathcal {M}}_{0^+ }^2)^{(0)}$ by an orthogonal matrix of

$$\begin{aligned} {\widetilde{U}}^{(0)}= & {} \left( \begin{array}{ccc} {\mathbb {I}}_{2\times 2} &{} &{} \\ &{}\quad c_\alpha &{}\quad - s_\alpha \\ &{}\quad s_\alpha &{}\quad c_\alpha \\ \end{array} \right) , \end{aligned}$$

(44)

into

$$\begin{aligned}{} & {} {\widetilde{U}}^{(0)} \cdot ( {\mathcal {M}}_{0^+ }^2)^{(0)} \cdot {\widetilde{U}}^{(0)\, T} = \textrm{diag}( 0, M_{\phi ^0 \phi ^0 }^2, {\widetilde{M}}_{h_1^\prime h_1^\prime }^{ 2}, {\widetilde{M}}_{h_2^\prime h_2^\prime }^{ 2} ),\nonumber \\ \end{aligned}$$

(45)

where $t_{\alpha }\!=\![ M_{h_1^\prime h_1^\prime }^2-\! M_{h_2^\prime h_2^\prime }^2 \!-\!\sqrt{( M_{h_1^\prime h_1^\prime }^2 -\! M_{h_2^\prime h_2^\prime }^2 )^2 \!+\! 4 M_{h_1^\prime h_2^\prime }^4}]/(2M_{h_1^\prime h_2^\prime }^2)$. The mixing matrix of ${\widetilde{U}}$ for the higher-order terms can be expanded up to ${\mathcal {O}}(\epsilon ^2)$ as

$$\begin{aligned} {\widetilde{U}}= & {} {\mathbb {I}}_{4\times 4} + {\widetilde{U}}^{(1)} + {\widetilde{U}}^{ (2)} + {\mathcal {O}}(\epsilon ^3), \end{aligned}$$

(46)

with ${\widetilde{U}}^{(1)} \sim {\mathcal {O}}(\epsilon )$ and ${\widetilde{U}}^{(2)} \sim {\mathcal {O}}(\epsilon ^2)$. Similarly, $V_{0^+}$ can also be expanded as:

$$\begin{aligned} V_{0^+}= & {} {\widetilde{U}}^{(0)} + V_{0^+}^{(1)} + V_{0^+}^{(2)} + {\mathcal {O}}(\epsilon ^3). \end{aligned}$$

(47)

For our later usage, we find that the $V_{0^+}^{(1)}$ is expressed as follows

$$\begin{aligned} V_{0^+}^{(1)}= & {} \left( \begin{array}{cccc} &{} - \frac{ M_{ h_u \phi ^0}^2 }{M_{\phi ^0 \phi ^0 }^2 } &{}\quad - \frac{ M_{ h_u h_1^\prime }^{ 2} }{M_{ h_1^\prime h_1^\prime }^{2 }} &{}\quad - \frac{ M_{ h_u h_2^\prime }^{2 } }{M_{ h_2^\prime h_2^\prime }^{ 2} } \\ \frac{ M_{ h_u \phi ^0 }^2 }{M_{\phi ^0 \phi ^0 }^2 } &{} &{} &{} \\ \frac{ M_{ h_u h_1^\prime }^{2 } }{M_{ h_1^\prime h_1^\prime }^{ 2} } &{} &{} &{} \\ \frac{ M_{ h_u h_2^\prime }^{2 } }{M_{ h_2^\prime h_2^\prime }^{ 2} } &{} &{} &{} \\ \end{array} \right) \nonumber \\\sim & {} \left( \begin{array}{cccc} &{} - \epsilon &{}\quad - \epsilon &{}\quad - \epsilon \\ \epsilon &{} &{} &{} \\ \epsilon &{} &{} &{} \\ \epsilon &{} &{} &{} \\ \end{array} \right) , \end{aligned}$$

(48)

with $(0 , 0, {\widetilde{M}}_{h_u h_1^\prime }^{2 }, {\widetilde{M}}_{h_u h_2^\prime }^{ 2} ) = {\widetilde{U}}^{(0)} \cdot ( 0 , 0, M_{ h_u h_1^\prime }^2 , M_{h_u h_2^\prime }^2 )$. By using the perturbation expansion in Eq. (47), we find the SM-like CP-even Higgs boson mass of

$$\begin{aligned} M_{H_u}^2= & {} \left( 2 \uplambda - \frac{ \nu v_\phi V_{331} }{ \sqrt{2} v_u^3 } \right) v_u^2 - \left[ (V_{0^+}^{(1)} )_{12}^2 M_{\phi ^0 \phi ^0 }^2 \right] \nonumber \\{} & {} - \left[ (V_{0^+}^{(1)} )_{13}^2 M_{h_1^\prime h_1^\prime }^2 \right] -\left[ (V_{0^+}^{(1)} )_{14}^2 M_{h_2^\prime h_2^\prime }^2 \right] . \end{aligned}$$

(49)

Since the mixing elements are $(V_{0^+}^{(1)})_{ij}\sim {\mathcal {O}}(\epsilon )$, all terms here are of the EW scales.

3.4 Summary of the Higgs spectrum

Table 1 The Higgs spectrum and the parameters in the Higgs potential

In the end of this section, we briefly summarize the Higgs spectrum in the current context. The symmetry breaking of ${\mathcal {G}}_{331}\rightarrow {\mathcal {G}}_{\textrm{SM}}$ and the sequential EWSB require eight NGBs, while the Higgs sector contains three $\textrm{SU}(3)_W$ anti-fundamentals of $\Phi _{\mathbf {\overline{3}},\rho }$ and $\Phi _{\mathbf {\overline{3}}}^\prime$. Therefore, we have ten real scalars in all. Through the above analysis, we find the 331 Higgs spectrum is consist of: two charged Higgs bosons of $H_{1,2}^\pm$ from Eq. (35), two CP-odd Higgs bosons of $(A^0, A^{0\,\prime })$ from Eq. (38), and four CP-even Higgs bosons of $(H_u, H_\phi , H_1, H_2)$ from Eqs. (40) and (41). The explicit expressions for Higgs mass matrix in Eqs. (32), (36), and (40) will be given in Appendix A.3. At the EW scale, our Higgs spectrum only contains one CP-even Higgs boson of $H_u$, while all other Higgs boson masses are decoupled. Therefore, our effective theory at the EW scale is distinct from the 2HDM, where a total of four Higgs bosons with masses of the EW scale are generally expected. Here, we list two benchmark models for the Higgs spectrum in Table 1 to demonstrate our results explicitly.

4 Bottom quark and tau lepton masses in the $\textrm{SU}(6)$

4.1 The Yukawa couplings

By taking the Higgs VEVs in Eqs. (21) and (24), we have the following mass matrices for the down-type (b, B) quarks and charged $(\tau ,E)$ leptons

$$\begin{aligned}{} & {} (Y_{\mathcal {D}})_{\rho \sigma } \left[ ( {\textbf{3}}, {\textbf{3}}, 0 )_{ {\textbf{F}}} \otimes \left( {\textbf{1}}, \bar{ {\textbf{3}}}, -\frac{1}{3}\right) _{ {\textbf{H}}}^\rho \otimes \left( \bar{{\textbf{3}}}, {\textbf{1}}, +\frac{1}{3}\right) _{ {\textbf{F}}}^{ \sigma }\right. \nonumber \\{} & {} \quad \oplus \left. \left( {\textbf{1}}, \bar{{\textbf{3}}}, +\frac{2}{3}\right) _{ {\textbf{F}}} \otimes \left( {\textbf{1}}, \bar{ {\textbf{3}}}, -\frac{1}{3} \right) _{ {\textbf{H}}}^\rho \otimes \left( {\textbf{1}}, \bar{{\textbf{3}}}, -\frac{1}{3} \right) _{ {\textbf{F}}}^{ \sigma } \right] \nonumber \\{} & {} \quad + H.c. \Rightarrow ({\bar{b}}_L, {\bar{B}}_L ) \cdot {\mathcal {M}}_{\mathcal {B}}\cdot \left( \begin{array}{c} b_R \\ B_R \\ \end{array} \right) + ({\bar{\tau }}_L, {\bar{E}}_L )\nonumber \\{} & {} \quad \cdot {\mathcal {M}}_{\mathcal {E}}\cdot \left( \begin{array}{c} \tau _R \\ E_R \\ \end{array} \right) + H.c., \end{aligned}$$

(50a)

$$\begin{aligned} {\mathcal {M}}_{\mathcal {B}}= & {} \frac{1}{ \sqrt{2} } \left( \begin{array}{cc} (Y_{\mathcal {D}})_{11} u_1 + ( Y_{\mathcal {D}})_{12} u_2 &{}\quad (Y_{\mathcal {D}})_{21} u_1 + ( Y_{\mathcal {D}})_{22} u_2 \\ (Y_{\mathcal {D}})_{11} V_1 + ( Y_{\mathcal {D}})_{12} V_2 &{} \quad (Y_{\mathcal {D}})_{21} V_1 + ( Y_{\mathcal {D}})_{22} V_2 \\ \end{array} \right) \nonumber \\{} & {} \quad \sim \left( \begin{array}{cc} \epsilon ^2 &{}\quad \epsilon ^2 \\ 1 &{}\quad 1 \\ \end{array} \right) V_{331}, \end{aligned}$$

(50b)

$$\begin{aligned} {\mathcal {M}}_{\mathcal {E}}= & {} \frac{1}{ \sqrt{2} } \left( \begin{array}{cc} (Y_{\mathcal {D}})_{11} u_1 + ( Y_{\mathcal {D}})_{12} u_2 &{}\quad - (Y_{\mathcal {D}})_{11} V_1 - ( Y_{\mathcal {D}})_{12} V_2 \\ (Y_{\mathcal {D}})_{21} u_1 + ( Y_{\mathcal {D}})_{22} u_2 &{}\quad -(Y_{\mathcal {D}})_{21} V_1 - ( Y_{\mathcal {D}})_{22} V_2 \\ \end{array} \right) \nonumber \\{} & {} \quad \sim \left( \begin{array}{cc} \epsilon ^2 &{}\quad -1 \\ \epsilon ^2 &{}\quad - 1 \\ \end{array} \right) V_{331}. \end{aligned}$$

(50c)

Given the seesaw-like mass matrices according to the mass hierarchy given in Eq. (31), a suppressed bottom quark and tau lepton masses of $\sim {\mathcal {O}}(1)\,\textrm{GeV}$ can be realized with the natural Yukawa couplings of $(Y_{{\mathcal {D}}})_{ij} \sim {\mathcal {O}}(1)$.

4.2 The bottom quark mass

Specifically, we first illustrate the bottom quark mass, and the tau lepton mass can be obtained straightforwardly. In general, the mass matrix in Eq. (50b) can be diagonalized through the bi-unitary transformation as

$$\begin{aligned}{} & {} U_L^{\mathcal {B}}{\mathcal {M}}_{\mathcal {B}}(U_R^{\mathcal {B}})^\dag = \left( \begin{array}{cc} m_b &{}\quad 0 \\ 0 &{}\quad m_B \\ \end{array} \right) ,\nonumber \\{} & {} U_{L/R}^{\mathcal {B}}= \left( \begin{array}{cc} c_{L/R} &{}\quad -s_{L/R} \\ s_{L/R} &{}\quad c_{L/R} \\ \end{array} \right) , \quad \left( \begin{array}{c} b_{L/R}^\prime \\ B_{L/R}^\prime \end{array} \right) = U_{L/R}^{\mathcal {B}}\cdot \left( \begin{array}{c} b_{L/R} \\ B_{L/R} \end{array} \right) , \end{aligned}$$

(51)

with $(b^\prime , B^\prime )$ being the mass eigenstates. We find that the corresponding Yukawa couplings are expressed in terms of masses and mixing angles as follows

$$\begin{aligned} (Y_{\mathcal {D}})_{11}= & {} \sqrt{2} \left[ ( c_L c_R m_b + s_L s_R m_B ) \frac{s_{{\tilde{\beta }}} }{ v_\phi } + ( - s_L c_R m_b \right. \nonumber \\{} & {} \quad \left. +c_L s_R m_B ) \frac{ c_{{\tilde{\beta }}} }{ V_{331} } \right] ,\end{aligned}$$

(52a)

$$\begin{aligned} (Y_{\mathcal {D}})_{12}= & {} \sqrt{2} \left[ ( c_L c_R m_b + s_L s_R m_B ) \frac{ -c_{{\tilde{\beta }}} }{ v_\phi } + ( - s_L c_R m_b \right. \nonumber \\{} & {} \quad \left. + c_L s_R m_B ) \frac{ s_{{\tilde{\beta }}} }{ V_{331} } \right] ,\end{aligned}$$

(52b)

$$\begin{aligned} (Y_{\mathcal {D}})_{21}= & {} \sqrt{2} \left[ ( - c_L s_R m_b + s_L c_R m_B ) \frac{s_{{\tilde{\beta }}} }{ v_\phi } + ( s_L s_R m_b\right. \nonumber \\{} & {} \quad \left. + c_L c_R m_B ) \frac{ c_{{\tilde{\beta }}} }{ V_{331} } \right] ,\end{aligned}$$

(52c)

$$\begin{aligned} (Y_{\mathcal {D}})_{22}= & {} \sqrt{2} \left[ ( - c_L s_R m_b + s_L c_R m_B ) \frac{ -c_{{\tilde{\beta }}} }{ v_\phi } + ( s_L s_R m_b \right. \nonumber \\{} & {} \quad \left. + c_L c_R m_B ) \frac{ s_{{\tilde{\beta }}}}{ V_{331} } \right] . \end{aligned}$$

(52d)

Under the reasonable limit of $\varphi _{L/R}\rightarrow 0$ and $t_{{\tilde{\beta }}}\sim 1$, we find the natural Yukawa couplings of $(Y_{{\mathcal {D}}})_{11}\sim (Y_{{\mathcal {D}}})_{12} \sim m_b/v_\phi \sim {\mathcal {O}}(1)$ and $(Y_{{\mathcal {D}}})_{21}\sim (Y_{{\mathcal {D}}})_{22} \sim m_B/V_{331} \sim {\mathcal {O}}(1)$.

By performing the orthogonal transformation in Eq. (39), we find the following bottom quark Yukawa couplings with the CP-even Higgs bosons

(53)

from Eq. (83b), and with the approximation given under the decoupling limit of $v_\phi / V_{331} \rightarrow 0$. By using the orthogonal transformation to CP-even Higgs fields in Eq. (41), the bottom quark Yukawa coupling with the SM-like Higgs boson of $H_u$ reads

$$\begin{aligned}{} & {} -{\mathcal {L}}_Y[H_u]\supset {-} m_b \overline{b_L^\prime } b_R^\prime \left[ \left( c_L^2 \frac{ c_\theta }{ v_\phi } {+} s_L^2 \frac{ s_\theta }{ V_{331} } \right) (V_{0^{+}}^{(1)} )_{12}\right. \nonumber \\{} & {} \qquad {+} s_L c_L \left( \frac{c_\theta }{ v_\phi } {+} \frac{s_\theta }{ V_{331} } \right) (V_{0^{+}}^{(1)} )_{13} \nonumber \\{} & {} \qquad \left. {+} \left( c_L^2 \frac{ s_\theta }{ v_\phi } {-} s_L^2 \frac{ c_\theta }{ V_{331} } \right) (V_{0^+}^{(1)} )_{14} \right] H_u {+} H.c. \nonumber \\{} & {} \quad \approx {-} m_b \overline{b_L^\prime } b_R^\prime \left[ \frac{ c_L^2 }{ v_\phi } (V_{0^+}^{(1)} )_{12} {+} \frac{ s_L c_L }{ v_\phi } (V_{0^+}^{(1)} )_{13}\right. \nonumber \\{} & {} \qquad \left. {+} \frac{ c_L^2 {-} s_L^2 }{V_{331} } (V_{0^+}^{(1)} )_{14} \right] H_u {+} H.c. \nonumber \\{} & {} \quad \approx {-} m_b \overline{b_L^\prime } b_R^\prime \left[ \frac{ c_L^2 }{ v_\phi } (V_{0^+}^{(1)} )_{12} {+} \frac{ s_L c_L }{ v_\phi } (V_{0^+}^{(1)} )_{13} \right] H_u {+} H.c.,\nonumber \\ \end{aligned}$$

(54)

with the mixing matrices in Eqs. (41), (43), (48) for the CP-even Higgs bosons. Likewise, we find the Yukawa couplings of the SM-like Higgs boson of $H_u$ with the heavy $B^\prime$ quark as

$$\begin{aligned} -{\mathcal {L}}_Y[H_u]\approx & {} - m_B \overline{B_L^\prime } B_R^\prime \left[ s_L^2 c_\theta \frac{s_\beta ^2 + s_\beta c_\beta }{ v_\phi } (V_{0^+}^{(1)} )_{12}\right. \nonumber \\{} & {} \left. - s_L c_L \frac{c_\theta }{ v_\phi } (V_{0^+}^{(1)} )_{13} \right] H_u + H.c.. \end{aligned}$$

(55)

One can expect two constraints from the SM sector, namely,

${\mathfrak {X}}$:

the EW charged currents mediated by $W^\pm$,

${\mathfrak {Y}}$:

all SM-like Higgs boson couplings, including $H_u {\bar{f}} f$, $H_u VV$ ($V=W^\pm ,Z$), $H_u gg$, and $H_u \gamma \gamma$.

From the EW charged currents given in terms of the gauge eigenstates in Eq. (79), it is straightforward to find that $V_{tb} = c_L$. It is thus natural to take the limit of $c_L\rightarrow 1$, according to the measurement of the CKM mixing angle of $|V_{tb}| = 1.013\pm 0.030$ [71]. Under this limit, the SM-like Higgs boson couplings to the heavy $B^\prime$ quark vanishes in Eq. (55) as $\varphi _L\rightarrow 0$. Thus, the potential heavy $B^\prime$ quark contributions to the effective $H_u gg$ and $H_u \gamma \gamma$ couplings are vanishing in this limit. Let us return to the bottom quark Yukawa coupling in Eq. (54) when $\varphi _L \rightarrow 0$, it is further simplified to

$$\begin{aligned} -{\mathcal {L}}_Y[H_u]\approx & {} - \frac{ m_b }{ v_\phi } (V_{0^+}^{(1)} )_{12}\, \overline{b_L^\prime } b_R^\prime H_u + H.c.. \end{aligned}$$

(56)

By requiring that the tree-level $H_u {\bar{b}}^\prime b^\prime$ Yukawa coupling in Eq. (54) is the same as the SM prediction [72, 73], we find the relation of

$$\begin{aligned}{} & {} \frac{v_{\textrm{EW}} }{ v_\phi } (V_{0^+}^{(1)} )_{12} \approx 1 \Rightarrow \frac{ v_{\textrm{EW}}^2 }{ v_\phi V_{331} } \approx 1, \end{aligned}$$

(57)

with the mixing angle of $(V_{0^+}^{(1)} )_{12} \sim \epsilon = \frac{v_{\textrm{EW}}}{V_{331}}$ in Eq. (48). For simplicity, the sub-leading correction term suppressed by $1/V_{331}$ in Eq. (54) is neglected. Apparently, this relation leads to the natural new physics scale for the 331 symmetry of

$$\begin{aligned}{} & {} \boxed { V_{331} \sim {\mathcal {O}}(10)\,\textrm{TeV}}, \end{aligned}$$

(58)

with the reasonable choice of $v_\phi \sim {\mathcal {O}}(1)\,\textrm{GeV}$ for the bottom quark Yukawa coupling. This confirms our previous assumption of the benchmark parameter input in Eq. (31). We have also checked that a new physics scale of $V_{331}$ in Eq. (58) is even consistent with the most stringent limit to the rare flavor-changing lepton decay process of $\textrm{Br}(\mu \rightarrow e \gamma )$ [74] when generalizing to the three-generational case [45].

4.3 The tau lepton mass

The tau lepton mass and Yukawa couplings follow closely from the bottom quark case, and we present the discussion here for completeness. The general ${\mathcal {E}}=(\tau , E)$ mass matrix in Eq. (50c) is related to the ${\mathcal {B}}=(b,B)$ mass matrix in Eq. (50b) by

$$\begin{aligned} {\mathcal {M}}_{\mathcal {E}}= & {} {\mathcal {M}}_{\mathcal {B}}^T \cdot \sigma _3. \end{aligned}$$

(59)

It is straightforward to find that the bi-unitary transformation for the ${\mathcal {E}}=(\tau , E)$ is simply related to those for the ${\mathcal {B}}=(b,B)$ as below

$$\begin{aligned}{} & {} U_L^{\mathcal {E}}{\mathcal {M}}_{\mathcal {E}}(U_R^{\mathcal {E}})^\dag = \left( \begin{array}{cc} m_\tau &{} \quad 0 \\ 0 &{} \quad m_E \\ \end{array} \right) ,\quad \left( \begin{array}{c} \tau _{L/R}^\prime \\ E_{L/R}^\prime \end{array} \right) = U_{L/R}^{\mathcal {E}}\cdot \left( \begin{array}{c} \tau _{L/R} \\ E_{L/R} \end{array} \right) \nonumber \\{} & {} U_{L}^{\mathcal {E}}= \left( \begin{array}{cc} c_{R} &{}\quad -s_{R} \\ s_{R} &{}\quad c_{R} \\ \end{array} \right) , \quad U_{R}^{\mathcal {E}}= \left( \begin{array}{cc} c_{L} &{} \quad s_{L} \\ -s_{L} &{}\quad c_L \\ \end{array} \right) . \end{aligned}$$

(60)

Immediately, this leads a result of $s_R=0$ from leptonic sector of the EW charged currents in Eq. (79). Analogous to Eqs. (52), the Yukawa couplings can also be expressed as

$$\begin{aligned} (Y_{\mathcal {D}})_{11}= & {} \sqrt{2} \left[ ( c_L c_R m_\tau - s_L s_R m_E ) \frac{s_{{\tilde{\beta }}} }{ v_\phi }\right. \nonumber \\{} & {} \left. + ( s_L c_R m_\tau + c_L s_R m_E ) \frac{ - c_{{\tilde{\beta }}} }{ V_{331} } \right] , \end{aligned}$$

(61a)

$$\begin{aligned} (Y_{\mathcal {D}})_{12}= & {} \sqrt{2} \left[ ( c_L c_R m_\tau - s_L s_R m_E ) \frac{ -c_{{\tilde{\beta }}} }{ v_\phi }\right. \nonumber \\{} & {} \left. + ( s_L c_R m_\tau + c_L s_R m_E ) \frac{ -s_{{\tilde{\beta }}} }{ V_{331} } \right] ,\end{aligned}$$

(61b)

$$\begin{aligned} (Y_{\mathcal {D}})_{21}= & {} \sqrt{2} \left[ ( - c_L s_R m_\tau - s_L c_R m_E ) \frac{s_{{\tilde{\beta }}} }{ v_\phi } \right. \nonumber \\{} & {} \left. + ( s_L s_R m_\tau - c_L c_R m_E ) \frac{ c_{{\tilde{\beta }}} }{ V_{331} } \right] ,\end{aligned}$$

(61c)

$$\begin{aligned} (Y_{\mathcal {D}})_{22}= & {} \sqrt{2} \left[ ( c_L s_R m_\tau + s_L c_R m_E ) \frac{c_{{\tilde{\beta }}} }{ v_\phi } + ( s_L s_R m_\tau \right. \nonumber \\{} & {} \left. - c_L c_R m_E ) \frac{ s_{{\tilde{\beta }}}}{ V_{331} } \right] . \end{aligned}$$

(61d)

Obviously, Eqs. (52) and (61) lead to the degenerate fermion mass predictions of $m_b = m_\tau$ and $m_B = -m_E$.¹² Thus, the b-$\tau$ mass unification issue cannot be addressed at the tree level. Their mass splitting can be attributed to the renormalization group running. This was first discussed in the context of the Georgi–Glashow $\textrm{SU}(5)$ model [75]. However, results therein cannot be naively applied to the $(b,\tau )$ mass ratio in the non-minimal GUTs. To fully evaluate their mass splitting, we expect two prerequisites of: (i) evaluation of the intermediate symmetry breaking scales from an appropriate GUT group, and (ii) the identification of the SM fermion representations with the extended color and weak symmetries. Both are distinctive features of the non-minimal GUTs, and we defer to analyze the details in the future work. By performing the orthogonal transformation in Eq. (39), we find the tau lepton Yukawa coupling with the SM-like Higgs boson the same as that for the bottom quark case in Eq. (54), with $m_b \rightarrow m_\tau$. Therefore, the scale of $V_{331}$ in Eq. (58) can be similarly determined from the tau lepton, given the current LHC measurements of the $H_u \tau \tau$ coupling [76, 77].

4.4 The possible radiative mechanism

We comment on the possible radiative fermion mass generation in the current scenario, which was proposed and considered to produce fermion mass hierarchies in various context [78,79,80,81,82,83,84,85,86]. In such a paradigm, the general assumption is that some light fermion masses can be radiatively generated with vanishing tree-level masses. Specifically, we should check that whether the bottom quark and tau lepton masses can be generated with $m_b=0$ in Eq. (51) and $m_\tau =0$ in Eq. (60). Let us consider the ${\mathcal {B}}=(b, B)$ case without loss of generality, the Yukawa couplings are reduced to the following expressions

$$\begin{aligned} (Y_{\mathcal {D}})_{11}= & {} \frac{m_B}{ v_\phi } s_R ( s_L s_{{\tilde{\beta }}} + t_\theta c_L c_{{\tilde{\beta }}} ),\end{aligned}$$

(62a)

$$\begin{aligned} (Y_{\mathcal {D}})_{12}= & {} \frac{m_B}{ v_\phi } s_R ( t_\theta c_L s_{{\tilde{\beta }}} - s_L c_{{\tilde{\beta }}} ) = (Y_{\mathcal {D}})_{22} t_R,\end{aligned}$$

(62b)

$$\begin{aligned} (Y_{\mathcal {D}})_{21}= & {} \frac{m_B}{ v_\phi } c_R ( s_L s_{{\tilde{\beta }}} + t_\theta c_L c_{{\tilde{\beta }}} ) = (Y_{\mathcal {D}})_{11} / t_R,\end{aligned}$$

(62c)

$$\begin{aligned} (Y_{\mathcal {D}})_{22}= & {} \frac{m_B}{ v_\phi } c_R ( t_\theta c_L s_{{\tilde{\beta }}} - s_L c_{{\tilde{\beta }}} ), \end{aligned}$$

(62d)

under the vanishing tree-level mass of $m_b=0$. The bottom quark and its heavy partner B can be mediated through the flavor-changing neutral vector bosons of $(N_\mu , {\bar{N}}_\mu )$ as in Eq. (78a), while this only happens for the left-handed components. Thus, the neutral vector bosons of $(N_\mu , {\bar{N}}_\mu )$ cannot lead to a radiative mass terms as was suggested in Refs. [80, 81]. The remaining possibility may be due to the mediation from the Higgs sector, as in Ref. [84]. By taking the $m_b=0$ in Eqs. (83a) and (83b), the neutral Higgs bosons can only mediate the left-handed $b^\prime$ and right-handed $B^\prime$. Thus, it is impossible to generate a radiative mass term of $m_b^{\textrm{rad}} \overline{b_L^\prime } b_R^\prime + H.c.$ with a vanishing tree-level $m_b=0$. The same argument also applies to the ${\mathcal {E}}=(\tau , E)$ case with the $m_\tau =0$ limit.

5 Conclusions

In this work, we study the bottom quark and tau lepton mass generations in the framework of one-generational $\textrm{SU}(6)$ GUT. The symmetry breaking stage of ${\mathcal {G}}_{331}\rightarrow {\mathcal {G}}_{\textrm{SM}}$ is found to be general for more realistic non-minimal GUTs with $n_g=3$. A different assignment to the Higgs VEVs from the previous studies is considered so that the bottom quark and tau lepton can obtain tree-level masses with $\sim {\mathcal {O}}(1)$ Yukawa couplings. We consider this fermion-Higgs mismatching pattern to be general, such as in more realistic unified model with the $\textrm{SU}(8)$ symmetry. Thus, we prevent the pattern leading to multiple EW Higgs doublets, which is very problematic with the ongoing LHC searches. An automatically generated small Higgs VEVs of $\sim {\mathcal {O}}(1)\,\textrm{GeV}$ is found to be possible as long as a gauge-invariant $\nu$-term in the Higgs potential can be of $\sim {\mathcal {O}}(100)\,\textrm{GeV}$. Notice that this $\nu$-term is also invariant under the emergent global symmetry of Eq. (2), which emerges automatically from the anomaly-free condition. By requiring the Yukawa coupling of the SM-like Higgs boson to SM fermions of $y_f^{\textrm{SM}}=\frac{\sqrt{2} m_f}{v_{\textrm{EW}} }$, we find the 331 symmetry-breaking scale of $V_{331}\sim 10\,\textrm{TeV}$ in the current context. This was not mentioned in the previous context. With the distinct VEV assignments in Eqs. (21) and (24), we find a Higgs sector consisting of one single CP-even Higgs boson at the EW scale. All other Higgs bosons have masses of $\sim {{{\mathcal {O}}}}(V_{331})$, as we have described in Sect. 3. Therefore, the effective theory at the EW scale contains only one SM-like CP-even Higgs boson, and is not described by a 2HDM.

Historically, it was proposed that three-generational SM fermions may be embedded non-trivially in a non-minimal GUT [3]. Through our recent analysis [21, 87], we find that the $\textrm{SU}(8)$ GUT can be the minimal model that have three-generational SM fermions transform differently under the extended gauge symmetries beyond the EW scale. Through the current discussion, we wish to mention the relations between the $\textrm{SU}(6)$ toy model and the realistic $\textrm{SU}(8)$ model. First, the $\textrm{SU}(6)$ subgroup of the ${{{\mathcal {G}}}}_{331}$ can be generic in the context of the $\textrm{SU}(8)$ GUT, as was shown in Eq. (15). Therefore, the results such as the ${{{\mathcal {G}}}}_{331}$ gauge sector and part of the Higgs sector in the current discussion can become useful in the context of the $\textrm{SU}(8)$ model. Second, the symmetry breaking pattern can be generalized, where the seemingly unnatural $\nu$-term in Eq. (19d) that generates the EWSB VEVs for the $(b,\tau )$ masses are natural due to the gravitational effect in the $\textrm{SU}(8)$ model. This means a potential relation between the gravitational effect and the flavor sector, which was never mentioned in any previous GUT literature according to our knowledge. Since the one-generational $\textrm{SU}(6)$ GUT is a toy model, there are several issues beyond the scope of the current discussions. They include: (i) the b-$\tau$ mass unification, (ii) the three-generational SM fermion mixings. Furthermore, the SM fermions in the non-minimal GUTs are usually accompanied with heavy partners from the $\textrm{SU}(N)$ anti-fundamentals. They can be mediated through the heavy charged and/or neutral vector bosons as well as heavy Higgs bosons during the intermediate symmetry breaking stages of the non-minimal GUT symmetry. It is therefore necessary to carry out detailed analysis of their experimental implications in some rare flavor-changing processes. All these issues will be studied elsewhere when extending to more realistic non-minimal GUTs such as the $\textrm{SU}(8)$, where three-generational SM fermions are embedded non-trivially.

The gauge symmetry breaking in the 331 model

In this section, we summarize the necessary results of the gauge symmetry breaking of ${{{\mathcal {G}}}}_{331} \rightarrow {{{\mathcal {G}}}}_{\textrm{SM}}$ for the current discussions as well as for the future studies.

1.1 The 331 gauge bosons

The kinematic terms for the $\textrm{SU}(3)_W$ Higgs fields are

$$\begin{aligned} {\mathcal {L}}= & {} \sum _\rho |D_\mu \Phi _{ \mathbf {\overline{3}},\rho } |^2 + |D_\mu \Phi _{ \mathbf {\overline{3}} }^\prime |^2. \end{aligned}$$

(63)

Generically, the covariant derivative for the $\textrm{SU}(3)_W$ fundamental representation is defined according to the convention in Refs. [35, 44]¹³

$$\begin{aligned}{} & {} D_\mu \Phi _{ \textbf{3} } \equiv ( \partial _\mu - i g_{3L} A_\mu ^a \frac{\uplambda ^a}{2} - i g_X X {\mathbb {I}}_3 X_\mu ) \Phi _{ \textbf{3} }, \end{aligned}$$

(64)

with $\uplambda ^a\,(a=1,\ldots ,8)$ being the $\textrm{SU}(3)$ Gell–Mann matrices. For the $\textrm{SU}(3)_W$ anti-fundamental representation, the covariant is defined as

$$\begin{aligned}{} & {} D_\mu \Phi _{ \mathbf {\overline{3}} } \equiv \left( \partial _\mu + i g_{3L} A_\mu ^a \frac{( \uplambda ^a)^* }{2} - i g_X X {\mathbb {I}}_3 X_\mu \right) \Phi _{ \mathbf {\overline{3}} } \nonumber \\{} & {} \quad = \left( \partial _\mu + i g_{3L} A_\mu ^a \frac{( \uplambda ^a)^T }{2} - i g_X X {\mathbb {I}}_3 X_\mu \right) \Phi _{ \mathbf {\overline{3}} }, \end{aligned}$$

(65)

with the hermiticity of $(\uplambda ^a)^\dag = \uplambda ^a$. Note that the definitions in Eqs. (64) and (65) are applicable to the $\textrm{SU}(3)_W$ fermions.

Explicitly, we express the gauge fields in terms of a $3\times 3$ matrix as follows

$$\begin{aligned} g_{ 3 L } A_\mu ^a \frac{\uplambda ^a}{2} {+} g_X X {\mathbb {I}}_3 X_\mu = \small { \frac{ 1 }{2} \left( \begin{array}{ccc} g_{3 L} ( A_\mu ^3 {+} \frac{1}{\sqrt{3} } A_\mu ^8 ) {+} 2 g_X X X_\mu &{}\quad g_{3L} (A_\mu ^1 {-} i A_\mu ^2 ) &{} \quad g_{3L} (A_\mu ^4 {-} i A_\mu ^5 ) \\ g_{3L} (A_\mu ^1 {+} i A_\mu ^2 ) &{} \quad g_{3 L} ( {-} A_\mu ^3 {+} \frac{1}{\sqrt{3} } A_\mu ^8 ) + 2 g_X X X_\mu &{}\quad g_{3L} (A_\mu ^6 - i A_\mu ^7 ) \\ g_{3L} (A_\mu ^4 + i A_\mu ^5 ) &{}\quad g_{3L} (A_\mu ^6 + i A_\mu ^7 ) &{} \quad - \frac{ 2 g_{3 L}}{\sqrt{3} } A_\mu ^8 + 2 g_X X X_\mu \\ \end{array} \right) } \end{aligned}$$

(66)

One can identify the charged gauge bosons of $W_\mu ^\pm \equiv \frac{1}{ \sqrt{2} } (A_\mu ^1 \mp i A_\mu ^2 )$, $C_\mu ^\pm \equiv \frac{1}{ \sqrt{2} } (A_\mu ^4 \mp i A_\mu ^5 )$, and the neutral gauge bosons of $N_\mu \equiv \frac{1}{ \sqrt{2} } (A_\mu ^6 - i A_\mu ^7 )$, and ${\bar{N}}_\mu \equiv \frac{1}{ \sqrt{2} } (A_\mu ^6 + i A_\mu ^7 )$. The electric charges of gauge bosons can be obtained by the relation of $[{\hat{Q}}, A_\mu ^a \uplambda ^a ] = Q_A^{IJ} (A_\mu ^a \uplambda ^a)_{IJ}$, with $X=0$ (since the $\textrm{SU}(3)_W$ gauge bosons do not take the $\textrm{U}(1)_X$ charges) in the electric charge operator given in Eq. (6).

The charged and neutral 331-gauge boson mass squares at the tree level read

$$\begin{aligned}{} & {} m_{C_\mu ^\pm }^2 = m_{N_\mu , {\bar{N}}_\mu }^2 = \frac{1}{4} g_{3L}^2 V_{331}^2, \end{aligned}$$

(67)

with the VEV assignment in Eq. (21) for simplicity. The other neutral gauge boson is due to the linear combination of $(A^8_\mu , X_\mu )$, whose mass matrix is

$$\begin{aligned}{} & {} \frac{1}{2}\cdot \frac{ (V_{331} )^2 }{9} ( A^{8\, \mu }, X^\mu ) \left( \begin{array}{cc} 3 g_{3L}^2 &{}\quad - \sqrt{3} g_{3L} g_X \\ - \sqrt{3} g_{3L} g_X &{}\quad g_X^2 \\ \end{array} \right) \nonumber \\{} & {} \quad \cdot \left( \begin{array}{c} A^8_\mu \\ X_\mu \\ \end{array} \right) , \nonumber \\{} & {} \quad \Rightarrow m_{Z_\mu ^\prime }^2 = \frac{1}{9}( g_X^2 + 3 g_{3L}^2 ) (V_{331} )^2. \end{aligned}$$

(68)

It is straightforward to define a mixing angle $\theta _X$ for the 331 symmetry breaking as

$$\begin{aligned} \left( \begin{array}{c} Z^\prime _\mu \\ B_\mu \\ \end{array} \right)= & {} \left( \begin{array}{cc} c_{X} &{} \quad - s_{X} \\ s_{X} &{}\quad c_{X} \\ \end{array} \right) \cdot \left( \begin{array}{c} A^8_\mu \\ X_\mu \\ \end{array} \right) , \end{aligned}$$

(69)

with

$$\begin{aligned} t_X\equiv & {} \tan \theta _{X} = \frac{g_X }{ \sqrt{3} g_{3L} }. \end{aligned}$$

(70)

Thus, $Z^\prime _\mu$ and $B_\mu$ can be expressed in terms of $A_\mu ^8$ and $X_\mu$ as

$$\begin{aligned}{} & {} Z^\prime _\mu = \frac{ \sqrt{3} g_{3L} A_\mu ^8 - g_X X_\mu }{ \sqrt{ g_X^2 + 3 g_{3L}^2 }}, \quad B_\mu = \frac{ g_{X} A_\mu ^8 + \sqrt{3} g_{3L} X_\mu }{ \sqrt{ g_X^2 + 3 g_{3L}^2 } }.\qquad \end{aligned}$$

(71)

The $\textrm{U}(1)_Y$ coupling of $\alpha _Y$ is related to the $\textrm{SU}(3)_W \otimes \textrm{U}(1)_X$ couplings of $(\alpha _{3\,L}, \alpha _X )$ as

$$\begin{aligned}{} & {} \alpha _Y^{-1} = \frac{1}{3} \alpha _{3L }^{-1} + \alpha _X^{-1}, \quad \frac{1}{3} \alpha _{3L}^{-1} = \alpha _Y^{-1} s_{X}^2,\quad \alpha _X^{-1} = \alpha _Y^{-1} c_{X}^2.\nonumber \\ \end{aligned}$$

(72)

Correspondingly, the diagonal components of the $\textrm{SU}(3)_W \otimes \textrm{U}(1)_X$ covariant derivative in Eq. (66) become

$$\begin{aligned}{} & {} \frac{1}{2}\textrm{diag} \left( g_{3L} A_\mu ^3 + g_Y \left( \frac{1}{3} + 2X \right) B_\mu , - g_{3L} A_\mu ^3 \right. \nonumber \\{} & {} \quad \left. + g_Y (\frac{1}{3} + 2X ) B_\mu , g_Y \left( -\frac{2}{3} + 2X \right) B_\mu \right) \nonumber \\{} & {} \quad + \frac{g_Y}{6} \textrm{diag} \left( -6X t_{X} + \frac{1}{t_X}, -6X t_{X} + \frac{1}{t_X},\right. \nonumber \\{} & {} \left. \quad -6X t_{X} - \frac{2}{t_X} \right) Z^\prime _\mu . \end{aligned}$$

(73)

Clearly, the $A_\mu ^3$ and $B_\mu$ terms from first two components recover the covariant derivatives in the EW theory with $X=\frac{1}{3}$. The off-diagonal components in Eq. (66) become

$$\begin{aligned}{} & {} \frac{g_{3L} }{ \sqrt{2}} \left( \begin{array}{ccc} 0 &{}\quad W_\mu ^+ &{}\quad C_\mu ^+ \\ W_\mu ^- &{}\quad 0 &{}\quad N_\mu \\ C_\mu ^- &{}\quad {\bar{N}}_\mu &{} \quad 0 \\ \end{array} \right) . \end{aligned}$$

(74)

Below, we call five massive gauge bosons of

$$\begin{aligned}{} & {} \left\{ C_\mu ^\pm , N_\mu , {\bar{N}}_\mu , Z_\mu ^\prime \right\} \end{aligned}$$

(75)

at this stage of symmetry breaking as the 331 gauge bosons, while the remaining ones of $\Big \{ W_\mu ^\pm , A_\mu ^3 , B_\mu \Big \}$ are the usual EW gauge bosons.

1.2 The gauge couplings of fermions

The $\textrm{SU}(3)_W \otimes \textrm{U}(1)_X$ covariant derivatives for chiral fermions in Table 1 are expressed as follows in terms of gauge eigenstates

$$\begin{aligned}{} & {} i D_\mu {\mathcal {B}}_R^\rho \supset g_X \left( {-} \frac{1}{3} \right) X_\mu {\mathcal {B}}_R^\rho , \end{aligned}$$

(76a)

$$\begin{aligned}{} & {} i D_\mu {\mathcal {L}}_L^\rho \supset \left( {-} g_{3L} A_\mu ^a \frac{ (\uplambda ^a)^T }{2} {+} g_X X_\mu {\mathbb {I}}_3 \left( {-}\frac{1}{3}\right) \right) {\mathcal {L}}_L^\rho ,\end{aligned}$$

(76b)

$$\begin{aligned}{} & {} i D_\mu t_R \supset g_X \left( {+}\frac{2}{3}\right) X_\mu t_R,\end{aligned}$$

(76c)

$$\begin{aligned}{} & {} i D_\mu {\mathcal {E}}_R \supset \left( {+} g_{3L} A_\mu ^a \frac{ \uplambda ^a }{2} {-} g_X X_\mu {\mathbb {I}}_3 \frac{2}{3} \right) {\mathcal {E}}_R,\end{aligned}$$

(76d)

$$\begin{aligned}{} & {} i D_\mu {\mathcal {Q}}_L \supset g_{3L} A_\mu ^a \frac{ \uplambda ^a }{2} {\mathcal {Q}}_L. \end{aligned}$$

(76e)

Analogous to the SM, we should find the charged currents and neutral currents for the $\textrm{SU}(3)_W \otimes \textrm{U}(1)_X$ gauge bosons.

The flavor-changing $\textrm{SU}(3)_W \otimes \textrm{U}(1)_X$ charged currents are mediated by $C_\mu ^\pm$ as follows

$$\begin{aligned} {\mathcal {L}}_{ \textrm{SU}(3)_W}^{\textrm{CC}}= & {} \frac{g_{3L} }{ \sqrt{2} } \left[ {\overline{t}}_L \gamma ^\mu B_L + \overline{ N_R} \gamma ^\mu \tau _R - \overline{{\widetilde{N}}_L^1} \gamma ^\mu \tau _L\right. \nonumber \\{} & {} \left. - \overline{{\widetilde{N}}_L^2} \gamma ^\mu E_L \right] C_\mu ^+ + H.c.. \end{aligned}$$

(77)

The $\textrm{SU}(3)_W \otimes \textrm{U}(1)_X$ neutral currents contain both the flavor-changing components mediated by $(N_\mu , {\bar{N}}_\mu )$, and the flavor-conserving components mediated by $Z^\prime _\mu$. In the chiral basis, they read

Apparently, the EW charged currents should reproduce the SM case, which are

$$\begin{aligned} {\mathcal {L}}_{ \textrm{SU}(2)_W}^{ \textrm{CC} }= & {} \frac{g_{3L} }{ \sqrt{2} } \left[ {\overline{t}}_L \gamma ^\mu b_L + {{\overline{\tau }}}_L \gamma ^\mu \nu _L \right] W_\mu ^+ + H.c. \end{aligned}$$

(79)

1.3 The mass matrices of the Higgs bosons

The matrix elements for the charged Higgs bosons in Eq. (32) read

$$\begin{aligned} M_{\phi _1^+ \phi _1^- }^2= & {} \frac{v_u \left[ \sqrt{2} \nu \left( s_{{\tilde{\beta }}}^2 (v_{\phi }^2 - V_{331}^2) -v_{\phi }^2 \right) +\kappa _{3} V_{331} v_{\phi } v_u \right] }{2 V_{331} v_{\phi }}, \end{aligned}$$

(80a)

$$\begin{aligned} M_{\phi _1^\pm \phi _2^\mp }^2= & {} \frac{ \nu v_u s_{{\tilde{\beta }} } c_{{\tilde{\beta }} } (V_{331}^2 - v_{\phi }^2 ) }{ \sqrt{2} V_{331} v_{\phi }}, \end{aligned}$$

(80b)

$$\begin{aligned} M_{\phi _2^+ \phi _2^- }^2= & {} \frac{ v_u \left[ \sqrt{2} \nu \left( c_{{\tilde{\beta }}}^2 (v_{\phi }^2 - V_{331}^2) - v_{\phi }^2 \right) +\kappa _{4} V_{331} v_{\phi } v_u \right] }{2 V_{331} v_{\phi }}, \end{aligned}$$

(80c)

$$\begin{aligned} M_{\phi _1^\pm \chi _1^\mp }^2= & {} \frac{1}{2} s_{{\tilde{\beta }} } ( \kappa _{3} v_{\phi } v_u-\sqrt{2} \nu V_{331} ),\end{aligned}$$

(80d)

$$\begin{aligned} M_{\phi _1^\pm \chi _2^\mp }^2= & {} \frac{1}{2} c_{{\tilde{\beta }}} ( \kappa _{3} V_{331} v_u-\sqrt{2} \nu v_{\phi } ),\end{aligned}$$

(80e)

$$\begin{aligned} M_{\phi _2^\pm \chi _1^\mp }^2= & {} \frac{1}{2} c_{{\tilde{\beta }}} ( \sqrt{2} \nu V_{331} - \kappa _{4} v_{\phi } v_u ),\end{aligned}$$

(80f)

$$\begin{aligned} M_{\phi _2^\pm \chi _2^\mp }^2= & {} \frac{1}{2} s_{{\tilde{\beta }}} ( \kappa _{4} V_{331} v_u-\sqrt{2} \nu v_{\phi } ),\end{aligned}$$

(80g)

$$\begin{aligned} M_{\chi _1^+ \chi _1^- }^2= & {} \frac{ v_{\phi } (-\sqrt{2} \nu V_{331}+\kappa _{3} v_{\phi } v_u s_{{\tilde{\beta }}}^2+\kappa _{4} v_{\phi } v_u c_{{\tilde{\beta }}}^2 ) }{2 v_u},\end{aligned}$$

(80h)

$$\begin{aligned} M_{\chi _1^\pm \chi _2^\mp }^2= & {} \frac{1}{4} V_{331} v_{\phi } s_{2 {\tilde{\beta }}} (\kappa _{3} - \kappa _{4}),\end{aligned}$$

(80i)

$$\begin{aligned} M_{\chi _2^+ \chi _2^- }^2= & {} \frac{V_{331} (\kappa _{3} V_{331} v_u c_{{\tilde{\beta }}}^2 + \kappa _{4} V_{331} v_u s_{{\tilde{\beta }}}^2 - \sqrt{2} \nu v_{\phi } )}{2 v_u}. \end{aligned}$$

(80j)

The matrix elements for the CP-odd Higgs bosons in Eq. (36) read

$$\begin{aligned} M_{\pi _u \pi _u}^2= & {} -\frac{\nu V_{331} v_{\phi }}{\sqrt{2} v_u},\end{aligned}$$

(81a)

$$\begin{aligned} M_{\pi _u \eta _1}^2= & {} \frac{\nu V_{331} s_{{\tilde{\beta }}}}{\sqrt{2}},\end{aligned}$$

(81b)

$$\begin{aligned} M_{\pi _u \eta _2}^2= & {} -\frac{\nu V_{331} c_{{\tilde{\beta }}}}{\sqrt{2}},\end{aligned}$$

(81c)

$$\begin{aligned} M_{\pi _u \pi _1}^2= & {} \frac{\nu v_{\phi } c_{{\tilde{\beta }}}}{\sqrt{2}},\end{aligned}$$

(81d)

$$\begin{aligned} M_{\pi _u \pi _2}^2= & {} \frac{\nu v_{\phi } s_{{\tilde{\beta }}}}{\sqrt{2}},\end{aligned}$$

(81e)

$$\begin{aligned} M_{\eta _1 \eta _1}^2= & {} \frac{1}{2} v_{\phi }^2 c_{{\tilde{\beta }}}^2 (\uplambda _{4}-\uplambda _{5})-\frac{\nu v_u \left[ s_{{\tilde{\beta }}}^2 (V_{331}^2 - v_{\phi }^2 ) +v_{\phi }^2 \right] }{\sqrt{2} V_{331} v_{\phi }},\end{aligned}$$

(81f)

$$\begin{aligned} M_{\eta _1 \eta _2}^2= & {} \frac{s_{{\tilde{\beta }}} c_{{\tilde{\beta }}} \left[ \sqrt{2} \nu ( V_{331}^2 -v_{\phi }^2 ) v_u+V_{331} v_{\phi }^3 (\uplambda _{4} - \uplambda _{5}) \right] }{2 V_{331} v_{\phi }},\end{aligned}$$

(81g)

$$\begin{aligned} M_{\pi _1 \eta _1}^2= & {} \frac{1}{4} V_{331} v_{\phi } s_{2{\tilde{\beta }}} (\uplambda _{5} - \uplambda _{4}),\end{aligned}$$

(81h)

$$\begin{aligned} M_{\pi _2 \eta _1}^2= & {} \frac{1}{2} \left[ V_{331} v_{\phi } c_{{\tilde{\beta }}}^2 (\uplambda _{4}-\uplambda _{5})-\sqrt{2} \nu v_u \right] ,\end{aligned}$$

(81i)

$$\begin{aligned} M_{\eta _2 \eta _2}^2= & {} \frac{\nu v_u \left[ c_{{\tilde{\beta }}}^2 (v_{\phi }^2 - V_{331}^2 ) -v_{\phi }^2 \right] }{\sqrt{2} V_{331} v_{\phi }}+\frac{1}{2} v_{\phi }^2 s_{{\tilde{\beta }}}^2 (\uplambda _{4}-\uplambda _{5}),\end{aligned}$$

(81j)

$$\begin{aligned} M_{\pi _1 \eta _2}^2= & {} \frac{1}{2} V_{331} v_{\phi } s_{{\tilde{\beta }}}^2 (\uplambda _{5}-\uplambda _{4})+\frac{\nu v_u}{\sqrt{2}},\end{aligned}$$

(81k)

$$\begin{aligned} M_{\pi _2 \eta _2}^2= & {} \frac{1}{4} V_{331} v_{\phi } s_{2{\tilde{\beta }}} (\uplambda _{4}-\uplambda _{5}),\end{aligned}$$

(81l)

$$\begin{aligned} M_{\pi _1 \pi _1}^2= & {} \frac{1}{2} V_{331}^2 s_{{\tilde{\beta }}}^2 (\uplambda _{4}-\uplambda _{5})+\frac{\nu v_u \left[ s_{{\tilde{\beta }}}^2 (v_{\phi }^2 -V_{331}^2 ) -v_{\phi }^2 \right] }{\sqrt{2} V_{331} v_{\phi }},\end{aligned}$$

(81m)

$$\begin{aligned} M_{\pi _1 \pi _2}^2= & {} \frac{s_{{\tilde{\beta }}} c_{{\tilde{\beta }}} \left[ V_{331}^3 v_{\phi } (\uplambda _{5}-\uplambda _{4})+\sqrt{2} \nu ( V_{331}^2 - v_{\phi }^2 ) v_u \right] }{2 V_{331} v_{\phi }},\end{aligned}$$

(81n)

$$\begin{aligned} M_{\pi _2 \pi _2}^2= & {} \frac{1}{2} V_{331}^2 c_{{\tilde{\beta }}}^2 (\uplambda _{4}-\uplambda _{5})+\frac{\nu v_u \left[ c_{{\tilde{\beta }}}^2 (v_{\phi }^2 -V_{331}^2 ) -v_{\phi }^2 \right] }{\sqrt{2} V_{331} v_{\phi }}. \end{aligned}$$

(81o)

The matrix elements for the CP-even Higgs bosons in Eq. (40) read

$$\begin{aligned} M_{h_u h_u}^2= & {} 2 \uplambda v_u^2 - \frac{\nu V_{331} v_{\phi }}{\sqrt{2} v_u},\end{aligned}$$

(82a)

$$\begin{aligned} M_{h_u \phi ^0 }^2= & {} -\frac{\sqrt{2} \nu V_{331}^2+2 V_{331} v_{\phi } v_u (\kappa _{1}+\kappa _{2})+\sqrt{2} \nu v_{\phi }^2}{2 \sqrt{V_{331}^2+v_{\phi }^2}},\end{aligned}$$

(82b)

$$\begin{aligned} M_{\phi ^0 \phi ^0 }^2= & {} -\frac{1}{ V_{331} v_{\phi } \left( V_{331}^2+v_{\phi }^2\right) } \left[ - V_{331}^3 v_{\phi }^3 (\uplambda _{1}+\uplambda _{2}+2 \uplambda _{3})\right. \nonumber \\{} & {} \left. +\frac{\nu }{ \sqrt{2} } ( V_{331}^2 - v_{\phi }^2 )^2 v_u \right] ,\end{aligned}$$

(82c)

$$\begin{aligned} M_{h_u h_1^\prime }^2= & {} v_u s_{{\tilde{\beta }}} c_{{\tilde{\beta }}} (\kappa _{1}-\kappa _{2}) \sqrt{V_{331}^2+v_{\phi }^2},\end{aligned}$$

(82d)

$$\begin{aligned} M_{ h_u h_2^\prime }^2= & {} \frac{v_u \left[ s_{{\tilde{\beta }}}^2 (\kappa _{2} V_{331}^2-\kappa _{1} v_{\phi }^2 )+c_{{\tilde{\beta }}}^2 (\kappa _{1} V_{331}^2-\kappa _{2} v_{\phi }^2 ) \right] }{\sqrt{V_{331}^2+v_{\phi }^2}},\end{aligned}$$

(82e)

$$\begin{aligned} M_{\phi ^0 h_1^\prime }^2= & {} V_{331} v_{\phi } s_{{\tilde{\beta }}} c_{{\tilde{\beta }}} (\uplambda _{2}-\uplambda _{1}),\end{aligned}$$

(82f)

$$\begin{aligned} M_{\phi ^0 h_2^\prime }^2= & {} \frac{1}{2} \left[ V_{331} v_{\phi } c_{2{\tilde{\beta }}} (\uplambda _{2}-\uplambda _{1})\right. \nonumber \\{} & {} \left. - \frac{(V_{331}^2 -v_{\phi }^2 ) \left( V_{331} v_{\phi } (\uplambda _{1}+\uplambda _{2}+2\uplambda _{3})+2 \sqrt{2} \nu v_u\right) }{V_{331}^2+v_{\phi }^2} \right] ,\nonumber \\ \end{aligned}$$

(82g)

$$\begin{aligned} M_{h_1^\prime h_1^\prime }^2= & {} -\frac{1}{8 V_{331} v_{\phi }}\left( V_{331}^2+v_{\phi }^2\right) (V_{331} v_{\phi } c_{4{\tilde{\beta }}} (\uplambda _{1}+\uplambda _{2}\nonumber \\{} & {} -2 (\uplambda _{3}+\uplambda _{4}+\uplambda _{5})) \nonumber \\{} & {} - V_{331} v_{\phi } (\uplambda _{1}+\uplambda _{2}+2 (-\uplambda _{3}+\uplambda _{4}+\uplambda _{5}))+4 \sqrt{2} \nu v_u),\end{aligned}$$

(82h)

$$\begin{aligned} M_{h_1^\prime h_2^\prime }^2= & {} \frac{1}{4} s_{2{\tilde{\beta }}} \left[ c_{2{\tilde{\beta }}} \left( V_{331}^2+v_{\phi }^2\right) (\uplambda _{1}\right. \nonumber \\{} & {} \left. +\uplambda _{2}-2 (\uplambda _{3}+\uplambda _{4}+\uplambda _{5})) +(\uplambda _{1}-\uplambda _{2}) (V_{331}^2 -v_{\phi }^2 ) \right] ,\end{aligned}$$

(82i)

$$\begin{aligned} M_{h_2^\prime h_2^\prime }^2= & {} \frac{1}{8 \left( V_{331}^2+v_{\phi }^2\right) } \left[ (V_{331}^4 + v_{\phi }^4) (3\uplambda _{1}+3\uplambda _{2}\right. \nonumber \\{} & {} +2 (\uplambda _{3}+\uplambda _{4}+\uplambda _{5}))+c_{4{\tilde{\beta }}} \left( V_{331}^2+v_{\phi }^2\right) ^2 (\uplambda _{1}+\uplambda _{2}\nonumber \\{} & {} -2 (\uplambda _{3}+\uplambda _{4}+\uplambda _{5})) + 4 c_{2{\tilde{\beta }}} (\uplambda _{1}-\uplambda _{2}) \left( V_{331}^4-v_{\phi }^4\right) \nonumber \\{} & {} \left. - 2 V_{331}^2 v_{\phi }^2 (\uplambda _{1}+\uplambda _{2}+6 \uplambda _{3}-2 (\uplambda _{4}+\uplambda _{5})) - 16 \sqrt{2} \nu \right. \nonumber \\{} & {} \left. V_{331} v_{\phi } v_u) \right] . \end{aligned}$$

(82j)

1.4 The Yukawa couplings of fermions

In terms of the fermion mass eigenstates of $(b^\prime , B^\prime )$, the CP-odd and CP-even Higgs Yukawa couplings are expressed as follows

$$\begin{aligned}{} & {} -{\mathcal {L}}_Y^{{\mathcal {Q}}, 0^-}\supset i m_b \overline{b_L^\prime } b_R^\prime \left( \zeta _1 \eta _1 + \zeta _2 \eta _2 + \zeta _3 \pi _1 + \zeta _4 \pi _2 \right) \nonumber \\{} & {} + i m_B \overline{b_L^\prime } B_R^\prime \left( \zeta _1^\prime \eta _1 + \zeta _2^\prime \eta _2 + \zeta _3^\prime \pi _1 + \zeta _4^\prime \pi _2 \right) \nonumber \\{} & {} + i m_b \overline{B_L^\prime } b_R^\prime \left( -\zeta _3 \eta _1 -\zeta _4 \eta _2 + \zeta _1 \pi _1 + \zeta _2 \pi _2 \right) \nonumber \\{} & {} + i m_B \overline{B_L^\prime } B_R^\prime \left( -\zeta _3^\prime \eta _1 - \zeta _4^\prime \eta _2 + \zeta _1^\prime \pi _1 + \zeta _2^\prime \pi _2 \right) + H.c.,\nonumber \\ \end{aligned}$$

(83a)

$$\begin{aligned}{} & {} -{\mathcal {L}}_Y^{{\mathcal {Q}}, 0^+} \supset m_b \overline{b_L^\prime } b_R^\prime \left( \zeta _1 \phi _1 + \zeta _2 \phi _2 + \zeta _3 h_1 + \zeta _4 h_2 \right) \nonumber \\{} & {} + m_B \overline{b_L^\prime } B_R^\prime \left( \zeta _1^\prime \phi _1 + \zeta _2^\prime \phi _2 + \zeta _3^\prime h_1 + \zeta _4^\prime h_2 \right) \nonumber \\{} & {} + m_b \overline{B_L^\prime } b_R^\prime \left( -\zeta _3 \phi _1 - \zeta _4 \phi _2 + \zeta _1 h_1 + \zeta _2 h_2 \right) \nonumber \\{} & {} + m_B \overline{B_L^\prime } B_R^\prime \left( -\zeta _3^\prime \phi _1 - \zeta _4^\prime \phi _2 + \zeta _1^\prime h_1 + \zeta _2^\prime h_2 \right) + H.c.,\nonumber \\ \end{aligned}$$

(83b)

where we parametrize the couplings as follows

$$\begin{aligned}{} & {} \zeta _1= c_L^2 \frac{s_{{\tilde{\beta }}} }{ v_\phi } - s_L c_L \frac{c_{{\tilde{\beta }}} }{ V_{331} },\quad \zeta _1^\prime = s_L c_L \frac{s_{{\tilde{\beta }}} }{ v_\phi } + c_L^2 \frac{c_{{\tilde{\beta }}} }{ V_{331} },\nonumber \\ \end{aligned}$$

(84a)

$$\begin{aligned}{} & {} \zeta _2 = - c_L^2 \frac{ c_{{\tilde{\beta }}} }{ v_\phi } - s_L c_L \frac{s_{{\tilde{\beta }}} }{ V_{331} },\quad \zeta _2^\prime = - s_L c_L \frac{ c_{{\tilde{\beta }}} }{ v_\phi } + c_L^2 \frac{s_{{\tilde{\beta }}} }{ V_{331} },\nonumber \\ \end{aligned}$$

(84b)

$$\begin{aligned}{} & {} \zeta _3 = -s_L c_L \frac{s_{{\tilde{\beta }}} }{ v_\phi } + s_L^2 \frac{c_{{\tilde{\beta }}} }{ V_{331} },\quad \zeta _3^\prime = -s_L^2 \frac{s_{{\tilde{\beta }}} }{ v_\phi } - s_L c_L \frac{c_{{\tilde{\beta }}} }{ V_{331} },\nonumber \\ \end{aligned}$$

(84c)

$$\begin{aligned}{} & {} \zeta _4 = s_L c_L \frac{c_{{\tilde{\beta }}} }{ v_\phi } + s_L^2 \frac{ s_{{\tilde{\beta }}} }{ V_{331} },\quad \zeta _4^\prime = s_L^2 \frac{c_{{\tilde{\beta }}} }{ v_\phi } - s_L c_L \frac{ s_{{\tilde{\beta }}} }{ V_{331} }. \nonumber \\ \end{aligned}$$

(84d)

For the fermion mass eigenstates of $( \tau ^\prime , E^\prime )$, their Yukawa couplings can be obtained by replacing $(m_b, m_B)\rightarrow ( m_\tau , - m_E)$ in the above Eqs. (83a) and (83b).