1 Introduction

The top quark, as the heaviest elementary particle in the Standard Model (SM), can decay into other light quarks associated with gauge or scalar bosons. Among all the possible decay modes, the flavour-changing neutral-current (FCNC) decays of the top quark, which are absent at tree level and suppressed at loop level by the Glashow–Iliopoulos–Maiani (GIM) mechanism [1] in the SM, are very sensitive to contributions from new physics (NP) [2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36]. For the two-body FCNC top-quark decays \(t\rightarrow qV\) and \(t\rightarrow qh\), with \(q=c,u\), \(V=\gamma ,Z,g\), and h the SM Higgs, their branching ratios are predicted to be of the orders of \(10^{-17}\) to \(10^{-12}\) within the SM, but could be enhanced by several orders of magnitude in many NP scenarios (see, e.g., Refs. [35, 36] and references therein). Among the three-body loop-induced top-quark decays \(t \rightarrow cVV\) and \(t \rightarrow u_1 \bar{u}_2 u_2\) with \(u_i=u,c\), the decay \(t\rightarrow cgg\) has the largest branching ratio, reaching up to about \(1.02\times 10^{-9}\) within the SM [37,38,39]. Interestingly, it is found that the branching ratio of the three-body decay \(t \rightarrow cgg\) is about two orders of magnitude larger than that of the two-body decay \(t \rightarrow cg\) within the SM [39], a phenomenon that will be dubbed “higher-order dominance” and has also been revealed in the bottom- and charm-quark decays [40,41,42,43,44,45,46]. Although their SM rates are predicted to be far below the detectable level of present and future colliders, these rare FCNC top-quark decays have been widely studied in various NP models, and significant enhancements are observed relative to the SM expectations; the most prominent examples are the two-Higgs-doublet model (2HDM) [2,3,4,5,6,7,8,9,10,11,12,13,14] and the supersymmetry [15,16,17,18,19,20,21,22,23,24,25], within which these rare top-quark decays might be even detectable at the future colliders. Especially, it is interesting to check if the “higher-order dominance” phenomenon observed in \(t\rightarrow cg(g)\) decays within the SM [39] is still valid for a viable NP model.

Among the various NP scenarios, the 2HDM [47] provides a minimal extension of the SM scalar sector by adding a second scalar doublet with hypercharge \(Y=1/2\), and can naturally satisfy the electroweak precision tests, giving rise at the same time to a very rich phenomenology [48, 49]. In a generic 2HDM, the non-diagonal couplings of the neutral scalars to the SM fermions would unavoidably lead to non-vanishing tree-level FCNC interactions. To avoid these unwanted tree-level FCNC couplings, an ad-hoc discrete \(\mathcal {Z}_2\) symmetry is often imposed on the Yukawa sector of the model [50]. Depending on the \(\mathcal {Z}_2\) charge assignments to the scalars and fermions, this results in four types of 2HDMs (types I, II, X, Y) [48, 49] under the hypothesis of natural flavour conservation [50]. In the aligned 2HDM (A2HDM) [51], on the other hand, the absence of tree-level FCNC interactions is automatically guaranteed by assuming the alignment in flavour space of the two Yukawa matrices for each type of the right-handed fermions. Interestingly, the A2HDM can recover, as particular cases, all the known specific implementations of the conventional 2HDMs based on \(\mathcal {Z}_2\) symmetries. The model is also featured by possible new sources of CP violation beyond that of the Cabibbo–Kobayashi–Maskawa (CKM) quark-mixing matrix [52, 53]. Thus, the A2HDM has been widely studied from the various theoretical points of view [54,55,56,57,58,59], as well as for its phenomenologies in the low-energy flavour physics [60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75], the electric dipole moments of the leptons, the neutron and various atoms [76,77,78,79], the anomalous magnetic moment of the muon [80,81,82,83,84,85], and the high-energy collider physics [12, 86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101]. Recently, assuming that there are no new sources of CP violation beyond the CKM quark-mixing matrix, and taking into account the most constraining processes and observables mentioned above, together with the theoretical requirements of perturbativity and positivity of the scalar potential, a detailed global fit has been performed to the A2HDM [75].

In this paper, we shall perform a detailed study of the rare top-quark decay \(t \rightarrow cgg\) and its comparison with the two-body decay \(t \rightarrow cg\) in the A2HDM as well as in the four conventional 2HDMs with \(\mathcal {Z}_2\) symmetries. Taking into account the relevant constraints on the model parameters from a global fit obtained at the \(95.5\%\) confidence level (CL) [75], we find that the branching ratios of \(t \rightarrow cg\) and \(t \rightarrow cgg\) decays can reach up to \(3.36\times 10^{-9}\) and \(2.95\times 10^{-9}\) respectively, being therefore of the same order, in the A2HDM. This is obviously different from the SM case, where the predicted branching ratio of \(t \rightarrow cgg\) is about two orders of magnitude larger than that of \(t \rightarrow cg\) [39]. On the other hand, compared with the SM predictions, no significant enhancements are observed in the four conventional 2HDMs with \(\mathcal {Z}_2\) symmetries for the branching ratios of these two decays. Searches for the decay \(t\rightarrow cg\) have been performed at the Large Hadron Collider (LHC), with the upper limit on its branching ratio being of \(\mathcal {O}(10^{-4})\) [102, 103], which is still much higher than the model predictions. These rare FCNC decays will also be studied at the future experiments, such as the high-luminosity LHC (HL-LHC) [104,105,106,107,108] and the Future Circular Collider (FCC) [109,110,111,112]. For example, the \(95\%\)-CL limit on the branching ratio of \(t \rightarrow cg\) at the 100-\(\mathrm {TeV}\) FCC in hadron-hadron mode (FCC-hh) with an integrated luminosity of \(10~\text {ab}^{-1}\) is estimated to be of \(\mathcal {O}(10^{-7}\)\(10^{-8})\) [113, 114], which is at least one order of magnitude higher than the maximum value predicted in the A2HDM. As a consequence, the predicted branching ratios of \(t \rightarrow cg\) and \(t \rightarrow cgg\) decays in the A2HDM are still out of the expected sensitivities of the future HL-LHC and FCC-hh.

This paper is organized as follows. In Sect. 2, we recapitulate the A2HDM, focusing only on the scalar and Yukawa sectors that are most relevant to us. In Sect. 3, we present our calculation of the decay rates of \(t \rightarrow cg(g)\) decays, with the resulting R functions present in the \(t\rightarrow cgg\) decay supplemented in Appendix A. Our numerical results are then given in Sect. 4, where we show the maximum branching ratios of \(t\rightarrow cg(g)\) decays that can be reached in the A2HDM, and give also a simple analysis of these two decays in the four conventional 2HDMs with \(\mathcal {Z}_2\) symmetries. Our conclusion is finally made in Sect. 5.

2 Aligned two-Higgs-doublet model

As a simple extension of the SM, the 2HDM is invariant under the SM gauge group and includes, besides the SM matter and gauge fields, two complex scalar \(SU(2)_L\) doublets,

$$\begin{aligned} \phi _{a}=\mathrm {e}^{i \theta _{a}}\left[ \begin{array}{c} \phi _{a}^{+} \\ \frac{1}{\sqrt{2}}\left( v_{a}+\rho _{a}+i \eta _{a}\right) \end{array}\right] , \end{aligned}$$
(2.1)

with the weak hypercharge \(Y=1/2\). Here we have assumed that the vacuum of the theory respects the electromagnetic gauge symmetry, which guarantees the vacuum expectation values (vevs) of the two scalar fields \(\phi _{a}\) to be aligned in the \(SU(2)_L\) space. After performing further a global \(SU(2)_L\) transformation to put the non-zero vevs only in the lower (neutral) components of \(\phi _{a}\), which guarantees the vacuum to be neutral and not charge-breaking, we can bring the vevs of the two scalar fields to the forms given by \(\langle 0\vert \phi _a^T(x)\vert 0 \rangle =(0,v_a e^{i \theta _a}/\sqrt{2})\), where \(v_1\) and \(v_2\) are real and non-negative [115, 116]. However, as one global phase can always be rotated away through an appropriate \(U(1)_Y\) transformation, we can, without loss of generality, fix \(\theta _1 = 0\) and leave the relative phase \(\theta =\theta _2-\theta _1\) [51, 115, 116].

Performing further a global SU(2) transformation in the scalar space \((\phi _1,\phi _2)\), we can rotate the original scalar basis to the so-called Higgs basis [116,117,118,119],

$$\begin{aligned} \left( \begin{array}{c} \Phi _1 \\ -\Phi _2 \end{array} \right) \equiv \left( \begin{array}{cc} \cos \beta &{} \sin \beta \\ \sin \beta &{} -\cos \beta \end{array} \right) \, \left( \begin{array}{c} \phi _1 \\ \mathrm {e}^{-i\theta }\phi _2 \end{array} \right) , \end{aligned}$$
(2.2)

where the rotation angle \(\beta \) is defined by the ratio of the absolute values of the two neutral scalar field vevs, \(\tan \beta \equiv \frac{\left| \left\langle 0|\phi _{2}^{0}\right| 0 \rangle \right| }{\left| \left\langle 0|\phi _{1}^{0}\right| 0 \rangle \right| }=\frac{v_2}{v_1}\), and, by convention, its value is limited to the first quadrant due to \(v_1,v_2\ge 0\). In fact, both \(\tan \beta \) and the relative phase \(\theta \) are basis-dependent, and we can remove them entirely by transforming from a generic basis \((\phi _1,\phi _2)\) to the Higgs basis \((\Phi _1,\Phi _2)\) [116,117,118,119]. The true significance of \(\tan \beta \) emerges only in specialized versions of the 2HDM, such as the four conventional 2HDMs with \(\mathcal {Z}_2\) symmetries, where \(\tan \beta \) is promoted to a physical parameter. In this case, we can remove the relative phase \(\theta \) by making a phase rotation \(\phi _2\rightarrow \mathrm {e}^{-i\theta }\phi _2\), such that the two neutral scalar vevs are real and the CP invariance is not spontaneously broken [116,117,118,119]. In the Higgs basis, only the scalar doublet \(\Phi _1\) gets a non-zero vev, \(\langle 0|\Phi _1^T\left( x\right) |0\rangle = (0, v/\sqrt{2})\), with \(v=\sqrt{v_1^2+v_2^2}=1/\sqrt{\sqrt{2}G_F}\simeq 246.22\,\mathrm{GeV}\), and plays the role of the SM Higgs doublet. The two scalar doublets in the Higgs basis can now be parametrized, respectively, as [51]

$$\begin{aligned} \Phi _1= & {} \left[ \begin{array}{c} G^+ \\ \frac{1}{\sqrt{2}}\, (v+S_1+iG^0) \end{array} \right] ,\nonumber \\ \Phi _2= & {} \left[ \begin{array}{c} H^+ \\ \frac{1}{\sqrt{2}}\, (S_2+iS_3) \end{array} \right] , \end{aligned}$$
(2.3)

where \(G^\pm \) and \(G^0\) denote the electroweak Goldstone fields, and the remaining five physical degrees of freedom are given by the two charged fields \(H^\pm (x)\) and the three neutral ones \(\varphi ^0_i(x) =\{h(x), H(x), A(x)\}=\mathcal {R}_{ij}S_j\), where \(\mathcal {R}\) is the orthogonal matrix needed for diagonalizing the mass terms in the scalar potential [92]. Generally, none of these three neutral scalars can have a definite CP quantum number. Here, for simplicity, we shall assume a CP-conserving Higgs sector, as will be detailed in the next two subsections.

2.1 Scalar sector

In the Higgs basis, the most general scalar potential allowed by the electroweak gauge symmetry \(SU(2)_L\otimes U(1)_Y\) takes the form [116,117,118]:

$$\begin{aligned} V&= \mu _{1}\,\left( \Phi _{1}^{\dagger }\Phi _{1}\right) \,+\,\mu _{2}\,\left( \Phi _{2}^{\dagger }\Phi _{2}\right) \, +\,\left[ \mu _{3}\,\left( \Phi _{1}^{\dagger }\Phi _{2}\right) \,\right. \nonumber \\&\quad \left. +\,\mu _{3}^{*}\,\left( \Phi _{2}^{\dagger }\Phi _{1} \right) \right] \nonumber \\&\quad +\lambda _{1}\,\left( \Phi _{1}^{\dagger }\Phi _{1}\right) ^{2}\,+\,\lambda _{2}\,\left( \Phi _{2}^{\dagger }\Phi _{2}\right) ^{2}\,\nonumber \\&\quad +\,\lambda _{3}\,\left( \Phi _{1}^{\dagger }\Phi _{1}\right) \left( \Phi _{2}^{\dagger }\Phi _{2}\right) \, +\,\lambda _{4}\,\left( \Phi _{1}^{\dagger }\Phi _{2}\right) \left( \Phi _{2}^{\dagger }\Phi _{1}\right) \nonumber \\&\quad +\left[ \left( \lambda _{5}\,\Phi _{1}^{\dagger }\Phi _{2}\,+\,\lambda _{6}\,\Phi _{1}^{\dagger }\Phi _{1}\,+\,\lambda _{7}\, \Phi _{2}^{\dagger }\Phi _{2}\right) \right. \nonumber \\&\quad \times \left. \left( \Phi _{1}^{\dagger }\Phi _{2}\right) \,+\,\mathrm {h.c.}\right] . \end{aligned}$$
(2.4)

Due to Hermiticity of the scalar potential, the parameters \(\mu _{1,2}\) and \(\lambda _{1,2,3,4}\) are real, while \(\mu _3\) and \(\lambda _{5,6,7}\) could be generally complex. The minimization condition of the scalar potential imposes the relations

$$\begin{aligned} \mu _1= -\lambda _1\,v^2,\quad \mu _3 = -\frac{1}{2}\,\lambda _6\,v^2. \end{aligned}$$
(2.5)

Inserting Eq. (2.3) into Eq. (2.4) and imposing the minimization condition given by Eq. (2.5), we can get the charged-Higgs mass

$$\begin{aligned} m_{H^{\pm }}^2= \mu _2+\frac{1}{2}\lambda _3\,v^2, \end{aligned}$$
(2.6)

while the masses of the three neutral scalars can be obtained after diagonalizing the mass-squared matrix \(\mathcal {M}\) of the three neutral-scalar fields \(S_{1,2,3}\) by the orthogonal matrix \(\mathcal {R}\), i.e., \(\mathrm {diag}\left( m_h^2, m_H^2,m_A^2\right) =\mathcal {R}\,\mathcal {M}\,\mathcal {R}^T\) [92]. In the CP-conserving limit of the scalar potential, the parameters \(\lambda _{5,6,7}\) are all real and the three neutral scalars h, H, and A are therefore CP eigenstates. In this case, the CP-odd scalar A corresponds to the field \(S_3\), while the two CP-even scalars h and H are the orthogonal combinations of \(S_1\) and \(S_2\):

$$\begin{aligned} \left( \begin{array}{c} h\\ H \end{array} \right) \; = \; \left( \begin{array}{cc} \cos {\tilde{\alpha }} &{} \sin {\tilde{\alpha }} \\ -\sin {\tilde{\alpha }} &{} \cos {\tilde{\alpha }} \end{array}\right) \; \left( \begin{array}{c} S_1\\ S_2 \end{array}\right) , \end{aligned}$$
(2.7)

where the mixing angle \(\tilde{\alpha }\) is determined by

$$\begin{aligned} \tan {\tilde{\alpha }}\; =\; \frac{m_h^2 - 2\lambda _1 v^2}{v^2\lambda _6} \; =\; \frac{v^2\lambda _6}{2\lambda _1 v^2- m_H^2}. \end{aligned}$$
(2.8)

Note that one can always restrict \(\tilde{\alpha }\) within the range \(0 \leqslant \tilde{\alpha }< \pi \) by performing a phase redefinition of the two CP-even fields \(S_1\) and \(S_2\). In addition, the masses of h and H satisfy the relation \(m_H^2\sin ^2{\tilde{\alpha }}+m_h^2\cos ^2{\tilde{\alpha }}=2\lambda _1 v^2\) [93], which, together with our convention \(m_h \le m_H\), implies that \(2\lambda _1 v^2 - m_H^2\le 0\) and \(m_h^2-2\lambda _1 v^2\le 0\). Then, from Eq. (2.8), one can see that the sign of \(\tan {\tilde{\alpha }}\) is uniquely determined by that of the parameter \(\lambda _6\), since the vev \(v=\sqrt{v_1^2+v_2^2}\) is, by definition, always positive. As the Higgs basis of a CP-conserving 2HDM is defined up to a global rephasing of the second scalar doublet \(\Phi _2 \rightarrow \mathrm {e}^{i\eta } \Phi _2\), we can use this freedom to change the sign of \(\lambda _6\) [120]. Here, without loss of generality, we shall use the rephasing \(\Phi _2 \rightarrow - \Phi _2\) to fix \(\lambda _6 \leqslant 0\) so that \(\tan {\tilde{\alpha }}\ge 0\), which, together with \(0 \leqslant \tilde{\alpha }< \pi \), results in our final choice \(0 \leqslant \tilde{\alpha }\leqslant \pi /2\). The masses of the three neutral scalars are finally given, respectively, by

$$\begin{aligned} m_{h}^2&= \frac{1}{2}(\Sigma -\Delta ),\quad m_{H}^2= \frac{1}{2}(\Sigma +\Delta ),\nonumber \\ m_{A}^2&= m_{H^{\pm }}^2+v^2 \left( \frac{\lambda _4}{2}-\lambda _5\right) , \end{aligned}$$
(2.9)

with

$$\begin{aligned} \Sigma =m_{H^{\pm }}^2+v^2\,\left( 2\lambda _2+\frac{\lambda _4}{2}-\lambda _5\right) , \end{aligned}$$
(2.10)
$$\begin{aligned} \Delta =\sqrt{\left[ m_{A}^2+2v^2\,(\lambda _5-\lambda _1)\right] ^2+4v^4\,\lambda _6^{2}}. \end{aligned}$$
(2.11)

Here the convention \(m_h \leqslant m_H\) has been chosen, and the SM limit is recovered when \(\tilde{\alpha }=0\).

2.2 Yukawa sector

The Yukawa Lagrangian of a 2HDM in the Higgs basis can be written as [49, 51]

$$\begin{aligned} \mathcal {L}_{Y,\text {weak}}&= -\frac{\sqrt{2}}{v}\,\Big [\bar{Q}^\prime _L (M^\prime _d \Phi _1 + Y^\prime _d \Phi _2) d^\prime _R \nonumber \\&\quad + \bar{Q}^\prime _L (M^\prime _u {\tilde{\Phi }}_1 + Y^\prime _u {\tilde{\Phi }}_2) u^\prime _R \nonumber \\&\quad + \bar{L}^\prime _L (M^\prime _\ell \Phi _1 + Y^\prime _\ell \Phi _2) \ell ^\prime _R \Big ] + \mathrm {h.c.} \end{aligned}$$
(2.12)

Here, \(Q_L^\prime \) and \(L_L^\prime \) are the left-handed quark and lepton doublets, while \(u^\prime _R\), \(d^\prime _R\), \(\ell ^\prime _R\) denote the right-handed fermion singlets, all being given in the weak-interaction basis; \({\tilde{\Phi }}_a(x)\equiv i\sigma _2\Phi _a^{*}(x)\), with \(\sigma _2\) the Pauli matrix, are the charge-conjugated fields with hypercharge \(Y=-1/2\); \(M^{\prime }_f\) (\(f=u,d,\ell \)) denote the non-diagonal matrices that encode both the fermion masses and the Yukawa couplings of the scalar doublet \(\Phi _1\) to fermions, while \(Y^{\prime }_f\) characterize only the Yukawa couplings of the second doublet \(\Phi _2\) to fermions.

In general, the two matrices \(M^{\prime }_f\) and \(Y^{\prime }_f\) associated with the same type of right-handed fermion \(u^\prime _R\), \(d^\prime _R\), or \(\ell ^\prime _R\) in Eq. (2.12) cannot be simultaneously diagonalized in flavour space. As a consequence, in the fermion mass-eigenstate basis with diagonal mass matrices \(M_f=U_L^{f\dagger }M^{\prime }_fU_R^f\), the corresponding Yukawa matrices \(Y_f=U_L^{f\dagger }Y^{\prime }_fU_R^f\) remain still non-diagonal and hence give rise to non-vanishing tree-level FCNC interactions. These unwanted tree-level FCNC couplings can, however, be eliminated by requiring that the two matrices \(M^{\prime }_f\) and \(Y^{\prime }_f\) are aligned in flavour space [51]. This results in the following alignment relations between the mass and the Yukawa matrices [51]:

$$\begin{aligned} Y_{d,\ell }=\varsigma _{d,\ell } \, M_{d,\ell },\quad Y_u=\varsigma _u^* \, M_u, \end{aligned}$$
(2.13)

where \(\varsigma _f\) are arbitrary complex parameters and could introduce new sources of CP violation beyond that of the CKM quark-mixing matrix. They are also scalar-basis independent and satisfy universality among the three different generations [51].

In terms of the fermion mass eigenstates, \(f_{L,R}=U_{L,R}^{f\dagger }f_{L,R}^{\prime }\), and making use of the alignment conditions specified by Eq. (2.13), we can finally rewrite the Yukawa Lagrangian of the A2HDM as [51]

$$\begin{aligned}&\mathcal {L}_{Y,\text {mass}} =-i \frac{G^0}{v} \Big \{\bar{d}_{L} M_{d} d_{R}+\bar{u}_{R} M_{u}^{\dagger } u_{L}+\bar{\ell }_{L} M_{\ell } \ell _{R}\Big \} \nonumber \\&\quad -\left( 1+\frac{S_{1}}{v}\right) \Big \{\bar{u}_{L} M_{u} u_{R}+\bar{d}_{L} M_{d} d_{R}+\bar{\ell }_{L} M_{\ell } \ell _{R}\Big \} \nonumber \\&\quad -\frac{1}{v} \left( S_2+i S_3\right) \Big \{\bar{d}_{L} Y_{d} d_{R}+\bar{u}_{R} Y_{u}^{\dagger } u_{L}+\bar{\ell }_{L} Y_{\ell } \ell _{R}\Big \}\nonumber \\&\quad - \frac{\sqrt{2}}{v} G^{+} \Big \{\bar{u}_{L} V_{\mathrm {CKM}} M_{d} d_{R} -\bar{u}_{R} M_{u}^{\dagger } V_{\mathrm {CKM}} d_{L} +\bar{\nu }_{L} M_{\ell } \ell _{R} \Big \}\nonumber \\&\quad - \frac{\sqrt{2}}{v} H^{+} \Big \{\varsigma _d\,\bar{u}_{L} V_{\mathrm {CKM}} M_{d} d_{R} - \varsigma _u\,\bar{u}_{R} M_{u}^{\dagger } V_{\mathrm {CKM}} d_{L} \nonumber \\&\quad + \varsigma _\ell \,\bar{\nu }_{L} M_{\ell } \ell _{R}\Big \}+\mathrm {h.c.}, \end{aligned}$$
(2.14)

where \(V_{\text {CKM}}=U_L^{u\dagger }U_L^d\) is the usual CKM matrix [52, 53], and the diagonal fermion mass matrices are now given, respectively, by

$$\begin{aligned} M_{u}&=\mathrm {diag}\left( m_{u}, m_{c}, m_{t}\right) , \quad M_{d}=\mathrm {diag}\left( m_{d}, m_{s}, m_{b}\right) , \nonumber \\ M_{\ell }&=\mathrm {diag}\left( m_{e}, m_{\mu }, m_{\tau }\right) . \end{aligned}$$
(2.15)

For the rare FCNC decays \(t\rightarrow cg(g)\) considered here, besides the SM part, only the interactions of the charged scalars \(H^{\pm }\) with the quarks are involved, which means that only the alignment parameters \(\varsigma _{u,d}\) and the charged-Higgs mass \(m_{H^{\pm }}\) will be involved throughout this work.

Table 1 The one-to-one correspondence between the different choices of the alignment parameters \(\varsigma _f\) and the four conventional 2HDMs based on discrete \(\mathcal {Z}_2\) symmetries

As shown in Table 1, the four conventional 2HDMs [48, 49] based on discrete \(\mathcal {Z}_2\) symmetries can be easily recovered from the A2HDM by choosing particular values of the alignment parameters \(\varsigma _f\). As the parameter \(\varsigma _\ell \) is not involved in the rare decays \(t \rightarrow cg(g)\), we are actually left with only two cases, type-I (type-X) 2HDM with \(\varsigma _d=\varsigma _u=\cot {\beta }\), and type-II (type-Y) 2HDM with \(\varsigma _d=-1/\varsigma _u=-\tan {\beta }\). Numerical results of the branching ratios of \(t \rightarrow cg(g)\) decays in these two specific cases as well as their comparisons with the SM predictions and the results obtained in the A2HDM will also be discussed in Sect. 4.

3 Calculation of the decay rates

3.1 General remarks

The rare top-quark decays \(t \rightarrow qg(g)\), with \(q=u,c\), occur firstly at one-loop level within the SM. As such, the decay rates of these processes are not only suppressed by the loop factor, but also receive a strong CKM and GIM suppression relative to the dominant tree-level decay \(t \rightarrow b W^+\) [2, 27, 39]. Due to the additional charged-Higgs contributions at the loop level, however, these decay rates could be enhanced in the A2HDM as well as in the conventional 2HDMs with \(\mathcal {Z}_2\) symmetries [2, 3, 7]. Here we shall focus on the decays into a charm instead of an up quark, because the decay rates of \(t \rightarrow ug(g)\) are further suppressed by the ratio \(|V_{ub}/V_{cb}|^2 \simeq 0.0087\) compared to that of \(t \rightarrow cg(g)\), both within the SM as well as in the 2HDMs considered.

Within the SM, due to the hierarchy of the CKM matrix elements, \(|V_{tb}|\gg |V_{td}|, |V_{ts}|\), the total width of the top quark is dominated by the tree-level two-body decay \(t \rightarrow b W^+\), and we can, therefore, take the approximation \(\Gamma _\mathrm {tot}(t) \simeq \Gamma (t \rightarrow b W^+)\). Keeping only the dominant next-to-leading-order (NLO) QCD correction [121,122,123] while neglecting the W-boson width effect [121], the NLO electroweak [124, 125], as well as the next-to-next-to-leading-order QCD correction [126,127,128,129,130], all being only of a few percentages of the Born term [127], we can write the partial decay width of the channel \(t \rightarrow b W^+\) as [123, 125]

$$\begin{aligned}&\Gamma (t \rightarrow b W^+) = \Gamma _0(t \rightarrow b W^+) \left\{ 1+\frac{C_{F}\alpha _s}{2\pi }\right. \nonumber \\&\quad \times \left[ \,2\frac{(1-\beta _W^2)(2\beta _W^2-1)(\beta _W^2-2)}{\beta _W^4(3-2\beta _W^2)}\ln (1-\beta _W^2)\right. \nonumber \\&\quad \left. - \frac{9-4\beta _W^2}{3-2\beta _W^2}\ln \beta _W^2+2\mathrm {Li}_2(\beta _W^2) -2\mathrm {Li}_2(1-\beta _W^2)\right. \nonumber \\&\quad \left. \left. -\frac{6\beta _W^4-3\beta _W^2-8}{2\beta _W^2(3-2\beta _W^2)}-\pi ^2\,\right] \right\} , \end{aligned}$$
(3.1)

where \(C_F = 4/3\) is the colour factor, and \(\alpha _s=g_s^2/(4\pi )\) denotes the strong coupling constant. The parameter \(\beta _W = \sqrt{1-m_W^2/m_t^2}\), with \(m_W\) and \(m_t\) the W-boson and the top-quark mass respectively, is the velocity of \(W^+\) in the top-quark rest frame, and the dilogarithm function is defined by \(\text{ Li}_2(x)=-\int _0^x \text {ln}(1-t)/t\,dt\). The leading-order (LO) decay width \(\Gamma _0(t \rightarrow b W^+)\) in Eq. (3.1) is given by [124]

$$\begin{aligned}&\Gamma _0(t \rightarrow b W^+) =\frac{G_F |V_{tb}|^2 \sqrt{\lambda (m_t,m_b,m_W)}}{8\pi \sqrt{2}\, m_t^3}\nonumber \\&\quad \times \Big [(m_t^2-m_b^2)^2+(m_t^2+m_b^2)m_W^2-2m_W^4\Big ], \end{aligned}$$
(3.2)

where \(G_F\) is the Fermi constant, \(m_b\) the bottom-quark mass, and \(\lambda (x,y,z)=(x^2-y^2-z^2)^2-4y^2z^2\) the usual triangle (or Källén) function for a two-body decay.

To a very good approximation, \(\Gamma _\mathrm {tot}(t) \simeq \Gamma (t \rightarrow b W^+)\) holds also in the A2HDM as well as in the four conventional 2HDMs with \(\mathcal {Z}_2\) symmetries, except when \(m_{H^{\pm }} < m_t - m_b\); in this regime, the top quark can have an appreciable chance to decay into the final state \(bH^+\), which must be taken into account for the total top width. In this case, we shall have \(\Gamma _\mathrm {tot}(t) \simeq \Gamma (t \rightarrow b W^+) + \Gamma (t \rightarrow b H^+)\), with the LO partial width of the decay \(t \rightarrow b H^+\) in the A2HDM given by [12]

$$\begin{aligned}&\Gamma (t \rightarrow bH^{+}) = \frac{G_F|V_{tb}|^2\sqrt{\lambda (m_t,m_b,m_{H^{\pm }})}}{8\pi \sqrt{2}\, m_t^3} \nonumber \\&\quad \times \Big [\left( m_t^2 + m_b^2 - m_{H^{\pm }}^2\right) \left( m_b^2|\varsigma _d|^2 + m_t^2|\varsigma _u|^2\right) \nonumber \\&\quad - 4 m_b^2 m_t^2 \,\mathrm {Re}(\varsigma _d \varsigma _u^*)\,\Big ]. \end{aligned}$$
(3.3)

As we are primarily interested in checking if the “higher-order dominance” phenomenon observed in \(t\rightarrow cg(g)\) decays within the SM [39] is still valid for the A2HDM and the four conventional 2HDMs with \(\mathcal {Z}_2\) symmetries, the charged-Higgs mass \(m_{H^{\pm }}\) will be assumed to be larger than the top-quark mass \(m_t\), and the decay mode \(t \rightarrow b H^+\) is therefore kinematically forbidden. Thus, the branching ratios of \(t \rightarrow cg(g)\) decays can be defined as

$$\begin{aligned} \mathcal {B}(t \rightarrow c g(g)) = \frac{\Gamma (t \rightarrow c g(g))}{\Gamma (t \rightarrow b W^+)}. \end{aligned}$$
(3.4)

In the next two subsections, we shall detail the calculation of the decay rates \(\Gamma (t \rightarrow c g(g))\) in the A2HDM. To this end, we shall follow the following procedure: Firstly, we implement the A2HDM into the Mathematica-based package FeynRules [131, 132] to generate the model file together with a complete set of Feynman rules, which is then fed into the package FeynArts [133, 134] to generate the one-loop Feynman diagrams and the corresponding amplitudes for the \(t\rightarrow cg(g)\) decays. Then, we resort to the packages FormCalc [135, 136] and LoopTools [135] to manipulate the decay amplitudes and evaluate numerically the scalar and tensor one-loop Feynman integrals. Meanwhile, some partial cross-checks have been done by using the package FeynCalc [137,138,139]. Finally, we can obtain the decay rates from the amplitudes squared by performing the phase space integration. Throughout this work, we carry out the one-loop calculation in \(D=4-2\epsilon \) space-time dimensions using the dimensional regularization scheme [140,141,142,143] for the ultraviolet divergence in loop integrals, and make the GIM mechanism manifest by dropping the terms independent of the internal down-type quark masses. For the collinear and infrared singularities present in the \(t\rightarrow cgg\) decay, we take a non-zero charm-quark mass to avoid the collinear divergence generated when one of the gluons in the final state travels parallelly to the charm quark, and introduce phase-space cuts to eliminate the collinear (generated when the two gluons travel parallelly to each other) and infrared (generated when one of the gluons is soft) divergences [39, 42].Footnote 1

As the two gluons are both on-shell and hence transversely polarized, care must be taken regarding the gluon polarization sums when squaring the amplitude of \(t\rightarrow cgg\) decay. One can maintain their transversality either by introducing contributions from the ghost fields while using the simple polarization sums [21, 39]Footnote 2

$$\begin{aligned}&\sum _{\text {polarizations}} \epsilon _{\mu }^{*}(k_3) \epsilon _{\nu }(k_3)\nonumber \\&\quad =\sum _{\text {polarizations}} \epsilon _{\mu }^{*}(k_4) \epsilon _{\nu }(k_4)=-g_{\mu \nu }, \end{aligned}$$
(3.5)

or via the explicit construction [40, 42]

$$\begin{aligned}&\sum _{\text {polarizations}} \epsilon _{\mu }^{*}(k_3) \epsilon _{\nu }(k_3)\nonumber \\&\quad =\sum _{\text {polarizations}} \epsilon _{\mu }^{*}(k_4) \epsilon _{\nu }(k_4)\nonumber \\&\quad =-g_{\mu \nu }+\frac{k_{3 \mu } k_{4 \nu }+k_{4 \mu } k_{3 \nu }}{k_{3} \cdot k_{4}}, \end{aligned}$$
(3.6)

which are now simultaneously transverse with respect to the two gluon momenta \(k_3\) and \(k_4\). Here, \(\epsilon _{\mu }(k_3)\) and \( \epsilon _{\mu }(k_4)\) denote the polarization vectors of the final-state gluons with momenta \(k_3\) and \(k_4\) respectively, while \(g_{\mu \nu }\) is the metric tensor in Minkowski space, with the Greek indices running over 0, 1, 2, 3. In this paper, we shall follow the second method. In practice, however, one needs only keep one of the two gluon polarization sums as in Eq. (3.6) while replacing the other one by \(-g_{\mu \nu }\) [40]. Conveniently, the package FormCalc provides already a function PolarizationSum to achieve such a task.

In our calculation, we shall use the ’t Hooft–Feynman gauge and take the external top and charm quarks to be on-shell. However, the gauge independence of our final results has been checked by performing additionally all the calculations in the unitary gauge. Due to the smallness of the down- and strange-quark masses \(m_{d,s}\) and the unitarity of the CKM matrix \(\sum _{q=d,s,b}V_{tq}^*V_{cq}=0\), the decay rates turn out to be only sensitive to the bottom-quark mass \(m_b\) [27, 35]. For these internal quark masses, the most adequate choice is the \(\overline{\mathrm{MS}}\) running mass evaluated at a typical scale of \(\mathcal {O}(m_t)\) [27, 35], and such a prescription will be followed by us. The external top- and charm-quark masses are, on the other hand, taken as the on-shell pole masses. For the sake of simplicity for our presentation, however, we shall set the charm-quark mass \(m_c\) to zero and keep only the contributions from the internal bottom quark to these decays, while taking into account all these effects in our numerical analyses presented in Sect. 4.

3.2 Two-body decay \(t \rightarrow cg\)

Let us firstly calculate the decay rate of the two-body decay \(t \rightarrow cg\) in the A2HDM. The one-loop Feynman diagrams responsible for this decay in the ’t Hooft–Feynman gauge are shown in Fig. 1, where the first three ones correspond to the vertex diagrams while the remaining ones to the flavour-changing (\(t-c\)) fermion self-energy diagrams. During the calculation of these Feynman diagrams, we have kept all the external and internal quark masses non-vanishing [2, 12]. Especially for the flavour-changing fermion self-energy diagrams, we have followed the treatment proposed in Ref. [146]: it is most straightforward to avoid the issue of counterterms [41, 42] altogether by simply calculating these diagrams, because the internal top (charm) quark line is off-shell and hence does not contribute to the one-particle pole of the charm (top) quark and needs not be truncated. Furthermore, due to the fact that the flavour-changing fermion self-energies are part of the non-truncated Green’s function, one must include their contributions with a factor of 1 rather than a factor of 1/2, as the latter applies only in the standard flavour-conserving case.

Fig. 1
figure 1

One-loop Feynman diagrams contributing to the \(t \rightarrow cg\) decay in the ’t Hooft–Feynman gauge, with \(d_i=\{d,s,b\}\), in the A2HDM. The first 1–3 diagrams are the vertex diagrams, and the rest 4–9 are the \(t-c\) fermion self-energy diagrams

The resulting transition amplitude for the two-body decay \(t \rightarrow cg\) can be written asFootnote 3

$$\begin{aligned}&\mathcal {A}_{t \rightarrow cg} = \frac{G_F}{4\sqrt{2}\,\pi ^2}\, g_s T^a_{ij} \sum _{q=d,s,b}V_{tq}^*V_{cq} \nonumber \\&\quad \times \left[ R_1^{(q)}(\epsilon ^*(k_3)\cdot k_1)\,\bar{c}_i(k_2,0)\, P_R\, t_j(k_1,m_t)\,\right. \nonumber \\&\quad \left. + R_2^{(q)}\,\bar{c}_i(k_2,0)\, P_R\, \epsilon \!\!/^*(k_3)\,t_j(k_1,m_t)\,\right] , \end{aligned}$$
(3.7)

where \(g_s=\sqrt{4\pi \alpha _s}\) is the strong coupling constant, \(T^a_{ij}\) the generators of the SU(3) gauge group with the indexes \(a=1,\ldots ,8\) and \(i,j=1,2,3\), and \(P_{R,L}=(1\pm \gamma _5)/2\) the chirality projectors. Here, \(\epsilon ^a_\mu (k_3)\) is the polarization vector of the gluon, while \(k_1\), \(k_2\), and \(k_3=k_1-k_2\) are the momenta of the top, the charm, and the gluon, respectively. The two form factors \(R_{1,2}^{(q)}\) in Eq. (3.7) are the sums of the contribution from each of the nine diagrams shown in Fig. 1. They can be expressed in terms of the masses of the external and internal quarks and bosons, and contain the D-dimensional loop integrals resulting from dimensional regularization. Specific to the internal bottom-quark contribution, we have explicitly

$$\begin{aligned} R_1^{(b)}&= 2m_t\Bigl \{2 m_W^2 C_{12}(0,m_t^2,0,m_b^2,m_b^2,m_W^2)\nonumber \\&\quad -(m_b^2-2m_W^2)\,C_{2}(0,m_t^2,0,m_b^2,m_b^2,m_W^2) \nonumber \\&\quad -m_b^2\bigl [C_0(0,m_t^2,0,m_b^2,m_b^2,m_W^2) \nonumber \\&\quad -C_{12}(0,m_t^2,0,m_b^2,m_b^2,m_W^2) \nonumber \\&\quad +\text {X}_{du}\left( C_0(0,m_t^2,0,m_b^2,m_b^2,m_{H^\pm }^2) \right. \nonumber \\&\quad \left. +C_{2}(0,m_t^2,0,m_b^2,m_b^2,m_{H^\pm }^2)\right) \nonumber \\&\quad -\text {X}_{dd}\,C_{12}(0,m_t^2,0,m_b^2,m_b^2,m_{H^\pm }^2)\bigr ] \Bigr \}, \end{aligned}$$
(3.8)

and

$$\begin{aligned} R_2^{(b)}&= m_W^2\Bigl \{2 m_t^2\big [C_{1}(0,m_t^2,0,m_b^2,m_b^2,m_W^2)\nonumber \\&\quad - m_b^2\,C_0(0,m_t^2,0,m_b^2,m_b^2,m_W^2)\nonumber \\&\quad -C_{2}(0,m_t^2,0,m_b^2,m_b^2,m_W^2)\big ] +2 B_{1}(m_t^2,m_b^2,m_W^2)\nonumber \\&\quad -4 C_{00}(0,m_t^2,0,m_b^2,m_b^2,m_W^2)\Bigr \} \nonumber \\&\quad + m_b^2\,\text {X}_{dd}\Bigl [(m_{H^\pm }^2-m_b^2)C_0(0,m_t^2,0,m_b^2,m_b^2,m_{H^\pm }^2)\nonumber \\&\quad -2 C_{00}(0,m_t^2,0,m_b^2,m_b^2,m_{H^\pm }^2)\nonumber \\&\quad + B_{1}(m_t^2,m_b^2,m_{H^\pm }^2)\Bigr ] \nonumber \\&\quad -\left( m_b^4-m_t^2 m_b^2-2 m_W^4\right) C_0(0,m_t^2,0,m_b^2,m_b^2,m_W^2) \nonumber \\&\quad + m_b^2\Bigl \{m_t^2\big [C_{1}(0,m_t^2,0,m_b^2,m_b^2,m_W^2) \nonumber \\&\quad +\text {X}_{dd}\,C_{1}(0,m_t^2,0,m_b^2,m_b^2,m_{H^\pm }^2) \nonumber \\&\quad +\text {X}_{du}\,C_0(0,m_t^2,0,m_b^2,m_b^2,m_{H^\pm }^2)\big ] \nonumber \\&\quad -\text {X}_{du}\big [B_{0}(0,m_b^2,m_{H^\pm }^2)+B_{0}(m_t^2,m_b^2,m_{H^\pm }^2)\big ] \nonumber \\&\quad -B_{0}(0,m_b^2,m_W^2)+B_{0}(m_t^2,m_b^2,m_W^2)\nonumber \\&\quad +B_{1}(m_t^2,m_b^2,m_W^2) \nonumber \\&\quad -2C_{00}(0,m_t^2,0,m_b^2,m_b^2,m_W^2)\Bigr \} \nonumber \\&\quad + \left[ (\text {X}_{dd}+1) m_b^2+2 m_W^2\right] B_{0}(0,m_b^2,m_b^2). \end{aligned}$$
(3.9)

Here, following the standard procedure and notation of Refs. [147,148,149,150], we have reduced the vector and tensor integrals via the Passarino–Veltman method [150] to the scalar one-loop integrals \(B_{l}\) and \(C_{l,lm}\), which can be further decomposed into the basic scalar functions \(A_0\), \(B_0\), and \(C_0\). However, as the package FormCalc does not require such a further reduction, we have expressed our results in terms of \(B_{l}\) and \(C_{l,lm}\) instead of the more basic ones to save CPU time. This is especially preferable for the three-body decay \(t \rightarrow c gg\), in which the more complicated four-point one-loop integrals \(D_{l,lm,lmn}\) will be involved. In addition, we have introduced the symbols \(\text {X}_{qq^{\prime }}=\varsigma _q\varsigma _{q^{\prime }}^*\), with \(q,q^{\prime }=d,u\), to represent the possible combinations of the alignment parameters in the A2HDM, and the SM result [2, 27] is reproduced by setting \(\text {X}_{qq^{\prime }}=0\). It should be noted that, although the individual contribution from each internal quark \(d_i\in \{d,s,b\}\) is separately divergent, the total amplitude is of course ultraviolet finite after summing over all the three internal quark flavours and taking into account the unitarity of the CKM matrix, \(\sum _{q=d,s,b}V_{tq}^*V_{cq}=0\).

With the transition amplitude at hand, it is straightforward to calculate the decay rate \(\Gamma (t \rightarrow cg)\), with the result given in the top-quark rest frame and in the limit \(m_c=0\) by

$$\begin{aligned}&\Gamma (t\rightarrow cg) = \frac{\sqrt{\lambda (m_t,m_c,0)}}{16 \pi m_t^3} \nonumber \\&\quad \times \sum _{\mathrm{{polarizations}}}\frac{1}{3}\sum _{\mathrm{{colours}}}\frac{1}{2}\sum _{\mathrm{{spins}}}\left| \mathcal {A}_{t \rightarrow cg}\right| ^2. \end{aligned}$$
(3.10)

For our later numerical analyses, however, we always take into account the finite charm-quark mass effect.

3.3 Three-body decay \(t \rightarrow cgg\)

In the A2HDM, the three-body decay \(t \rightarrow cgg\) receives contributions from the one-loop Feynman diagrams shown in Fig. 2, where \(d_i=\{d,s,b\}\) and the ’t Hooft–Feynman gauge is adopted. The transition amplitude of the decay can be written as

$$\begin{aligned} \mathcal {A}_{t \rightarrow cgg} \!=\! \sum _{q=d,s,b} \Big [\mathcal {A}_{(q)}^{\text {vertex}} + \mathcal {A}_{(q)}^{\text {box}} + \mathcal {A}_{(q)}^{\text {self}}\,\Big ], \end{aligned}$$
(3.11)

where \(\mathcal {A}_{(q)}^{\text {vertex}}\), \(\mathcal {A}_{(q)}^{\text {box}}\), and \(\mathcal {A}_{(q)}^{\text {self}}\) represent the contributions from the vertex, the box, and the self-energy diagrams, respectively. Specific again to the internal bottom-quark contribution and in the limit \(m_c=0\), these separate terms can be written in a compact form as

$$\begin{aligned} \mathcal {A}_{(b)}^{\text {vertex}}&=\frac{G_F\,\alpha _s V_{tb}^{*}V_{cb}}{\sqrt{2}\,\pi \, t_{12}\, s_{23}\, t\, R_1 R_2}\Big [\,T_d\,t_{12}\,t\,R_2\big (R_1R_1^{\text {vertex}}\nonumber \\&\quad +s_{23}R_2^{\text {vertex}}\big ) + T_e\,t_{12}\,s_{23}\,R_1\big (R_2R_3^{\text {vertex}} + t R_4^{\text {vertex}}\big ) \nonumber \\&\quad + \big (T_d-T_e\big ) s_{23}\,t\,R_1R_2R_5^{\text {vertex}}\,\Big ], \end{aligned}$$
(3.12)
$$\begin{aligned} \mathcal {A}_{(b)}^{\text {box}}&=\frac{G_F\,\alpha _s V_{tb}^{*}V_{cb}}{\sqrt{2}\,\pi \, t_{12}\, s_{23}\, t\, R_1 R_2}\Big [\,m_t\big (F_1 R_1^{\text {box}}+ F_{12}R_5^{\text {box}}\nonumber \\&\quad +2F_{13}R_6^{\text {box}}+2F_{14}R_7^{\text {box}}\big ) \nonumber \\&\quad - \big (F_{3}R_2^{\text {box}}+F_{4}R_3^{\text {box}}+F_{5}R_4^{\text {box}} -F_{16}R_8^{\text {box}}\big )\,\Big ], \end{aligned}$$
(3.13)
$$\begin{aligned} \mathcal {A}_{(b)}^{\text {self}}&=\frac{G_F\,\alpha _s V_{tb}^{*}V_{cb}}{\sqrt{2}\,\pi \, t_{12}\, s_{23}\, t\, R_1 R_2}\Big [\,T_d\,t_{12}\,t\,R_2\big (R_1R_1^{\text {self}}\nonumber \\&\quad +s_{23}\,m_b^2R_2^{\text {self}}+R_6^{\text {self}}\big ) \nonumber \\&\quad + T_e\,t_{12}\,s_{23}\,R_1\big (R_2R_3^{\text {self}} +t\,m_b^2R_4^{\text {self}}+R_7^{\text {self}}\big )\nonumber \\&\quad + \big (T_d-T_e\big ) s_{23}\,t\,R_1 R_2 R_5^{\text {self}}\,\Big ]. \end{aligned}$$
(3.14)

Here the kinematics of the process is specified in the parentheses of the initial- and final-state particles as \(t(k_1) \rightarrow c(k_2)+g(k_3)+g(k_4)\). For convenience, we have also introduced in Eqs. (3.12)–(3.14) two products of the colour generators, \(T_d=(T^aT^b)_{ij}\) and \(T_e=(T^bT^a)_{ij}\), as well as the (generalized) Mandelstam variables \(t=(k_{1}-k_{3})^2\), \(t_{12}=(k_{1}-k_{2})^2\), and \(s_{23}=(k_{2}+k_{3})^2\).Footnote 4 In addition, the independent fermion chains involved are defined, respectively, by

$$\begin{aligned} F_1&= \bar{c}_i(k_2,0)\, P_R \,t_j(k_1,m_t),\nonumber \\ F_2&= \bar{c}_i(k_2,0)\, P_L \,t_j(k_1,m_t),\nonumber \\ F_3&= \bar{c}_i(k_2,0)\, P_R \, \epsilon \!\!/^*(k_3)\,t_j(k_1,m_t),\nonumber \\ F_4&= \bar{c}_i(k_2,0)\, P_R\, \epsilon \!\!/^*(k_4)\, t_j(k_1,m_t),\nonumber \\ F_5&= \bar{c}_i(k_2,0)\, P_R\, k\!\!\!/_3 \,t_j(k_1,m_t), \nonumber \\ F_6&= \bar{c}_i(k_2,0)\, P_L\, \epsilon \!\!/^*(k_3) \,t_j(k_1,m_t), \nonumber \\ F_7&= \bar{c}_i(k_2,0)\, P_L\, \epsilon \!\!/^*(k_4) \,t_j(k_1,m_t),\nonumber \\ F_8&= \bar{c}_i(k_2,0)\, P_L\, k\!\!\!/_3 \,t_j(k_1,m_t), \nonumber \\ F_9&= \bar{c}_i(k_2,0)\, P_L\, \epsilon \!\!/^*(k_3)\, \epsilon \!\!/^*(k_4)\,t_j(k_1,m_t), \nonumber \\ F_{10}&= \bar{c}_i(k_2,0)\, P_L\, \epsilon \!\!/^*(k_3)\, k\!\!\!/_3 \,t_j(k_1,m_t),\nonumber \\ F_{11}&= \bar{c}_i(k_2,0)\, P_L\, \epsilon \!\!/^*(k_4)\, k\!\!\!/_3 \,t_j(k_1,m_t),\nonumber \\ F_{12}&= \bar{c}_i(k_2,0)\, P_R\, \epsilon \!\!/^*(k_3)\, \epsilon \!\!/^*(k_4) \,t_j(k_1,m_t), \nonumber \\ F_{13}&= \bar{c}_i(k_2,0)\, P_R\, \epsilon \!\!/^*(k_3)\, k\!\!\!/_3 \,t_j(k_1,m_t),\nonumber \\ F_{14}&= \bar{c}_i(k_2,0)\, P_R\, \epsilon \!\!/^*(k_4)\, k\!\!\!/_3 \,t_j(k_1,m_t),\nonumber \\ F_{15}&= \bar{c}_i(k_2,0)\, P_L\, \epsilon \!\!/^*(k_3)\, \epsilon \!\!/^*(k_4)\, k\!\!\!/_3 \,t_j(k_1,m_t),\nonumber \\ F_{16}&= \bar{c}_i(k_2,0)\, P_R\, \epsilon \!\!/^*(k_3)\, \epsilon \!\!/^*(k_4)\, k\!\!\!/_3 \,t_j(k_1,m_t). \end{aligned}$$
(3.15)

For the R functions present in Eqs. (3.12)–(3.14), we have \(R_1=s_{23}-m_t^2\) and \(R_2=t-m_t^2\), while the explicit expressions of all the remaining ones are collected in Appendix A.

Fig. 2
figure 2

One-loop Feynman diagrams contributing to the \(t \rightarrow cgg\) decay in the ’t Hooft–Feynman gauge, with \(d_i=\{d,s,b\}\), in the A2HDM. The first 1–15 diagrams are the vertex diagrams, the diagrams 16–21 are the box diagrams, and the rest 22–45 are the \(t-c\) fermion self-energy diagrams

In the top-quark rest frame, the differential decay rate of \(t \rightarrow cgg\) can be written as

$$\begin{aligned}&d\Gamma (t\rightarrow cgg) = \frac{1}{2m_t}\left( \prod _{i=2}^{4} \int \frac{d^{3} k_{i}}{(2 \pi )^{3}} \frac{1}{2 E_{i}}\right) (2 \pi )^{4} \delta ^{(4)}\nonumber \\&\quad \times \big (k_1-k_2-k_3-k_4\big )\nonumber \\&\quad \times \sum _\text {polarizations}\frac{1}{3}\sum _\text {colours}\frac{1}{2}\sum _\text {spins} \frac{1}{2} \left| \mathcal {A}_{t \rightarrow cgg}\right| ^2, \end{aligned}$$
(3.16)

where \(E_i=k_i^0\) represents the energy of the (i-1)-th final-state particle in the top-quark rest frame, and the last factor 1/2 accounts for the statistical factor due to the two gluons in the final state. In this frame, we have \(\vec {k}_2+\vec {k}_3+\vec {k}_4=0\), and thus the three final-state momentum vectors lie in a plane. Then, the three-body phase space in Eq. (3.16) can be integrated over the orientation of this plane by using the momentum delta functions and the azimuthal symmetry. In this way, we are left with only two non-trivial integration variables, which can be chosen as the energy of the charm quark (\(k_2^0\)) and the energy of one of the gluons (say \(k_3^0\)), and the decay width of \(t \rightarrow cgg\) can be finally written as

$$\begin{aligned}&\Gamma (t\rightarrow cgg) = \frac{1}{64\,\pi ^3\,m_t} \sum _\text {polarizations}\nonumber \\&\quad \times \frac{1}{3}\sum _\text {colours}\frac{1}{2}\sum _\text {spins} \int dk_3^0\int dk_2^0 \times \frac{1}{2} \left| \mathcal {A}_{t \rightarrow cgg}\right| ^2. \end{aligned}$$
(3.17)

Because of the infrared and collinear divergences mentioned in Sect. 3.1, which are also evident from Eqs. (3.12)–(3.14) and appear at the vertices and along the boundary lines of the Dalitz plot in the \((k_2^0, k_3^0)\) space respectively, we have to impose proper cuts on both \(k_2^0\) and \(k_3^0\) when performing the integration of Eq. (3.17) over these two variables. Following the treatment of Refs. [39, 42], we shall use the kinematic constraints

$$\begin{aligned}&t=(k_1-k_3)^2=m_{t}^{2} - 2 m_{t} k_{3}^0 \geqslant C \cdot 2 m_{t}^{2}, \nonumber \\&t_{12}=(k_1-k_2)^2=m_{t}^{2} - 2 m_{t} k_{2}^0 \geqslant C \cdot 2 m_{t}^{2}, \nonumber \\&s_{23}=(k_2+k_3)^2=(k_1-k_4)^2 \nonumber \\&=-m_{t}^{2} + 2 m_{t} (k_{2}^0 + k_{3}^0) \geqslant C \cdot 2 m_{t}^{2}, \end{aligned}$$
(3.18)

along with the fact that \(k_{2,\text {max}}^0 \geqslant k_{2,\text {min}}^0\), to get the integration ranges of the two variables:Footnote 5

$$\begin{aligned}&k_2^0 \in \left[ \frac{m_t}{2}-k_3^0+Cm_t,\frac{m_t}{2}(1-2C)\right] , \nonumber \\&k_3^0 \in \left[ 2Cm_t,\frac{m_t}{2}(1-2C)\right] , \end{aligned}$$
(3.19)

where C is the cutoff parameter, and its value must be taken within the range of jet energy resolution of a detector. Note that, when a non-zero charm-quark mass is considered, Eq. (3.19) must be modified as

$$\begin{aligned}&k_2^0 \in \left[ \frac{m_t}{2}-k_3^0+Cm_t+\frac{m_c^2}{2m_t},\frac{m_t}{2}(1-2C)+\frac{m_c^2}{2m_t}\right] , \nonumber \\&k_3^0 \in \left[ 2Cm_t,\frac{m_t}{2}(1-2C)-\frac{m_c^2}{2m_t}\right] . \end{aligned}$$
(3.20)

Here, for simplicity, we shall take \(C\simeq 0.01\) as the default value (i.e., requiring the energy of each decay product to be larger than \(2.0\,\mathrm{GeV}\) in the top-quark rest frame), while varying C within the range [0.001, 0.05] to study its numerical impacts on the branching ratio of \(t \rightarrow cgg\) decay. Note that the choice \(C=0.01\) is already large enough to reach the jet energy resolution sensitivity of the HL-LHC [108] and FCC [109,110,111,112] facilities.

4 Numerical results and discussions

4.1 Input parameters

The relevant input parameters needed for evaluating the branching ratios of \(t \rightarrow cg(g)\) decays are listed in Table 2. For the QCD and electroweak parameters as well as the quark masses, their values are taken from the Particle Data Group [151]. Here, for the external on-shell quarks, we take their pole masses as input, and evolve the internal quark masses from their respective initial scales to the scale \(\mu _t=m_t^\mathrm {pole}\) characteristic for the top-quark decay, using the two-loop renormalization group equations for the \(\overline{\mathrm{MS}}\) running quark masses.Footnote 6 The \(\overline{\mathrm{MS}}\) coupling constant \(\alpha _s\) is given at the Z-boson mass scale with the top quark decoupled (i.e., with \(n_f=5\) active quark flavours). To obtain its values at the scale \(\mu _t\) and also at other scales when evolving the \(\overline{\mathrm{MS}}\) running quark masses, we use the two-loop running of \(\alpha _s\) by taking into account the decoupling effect when crossing a flavour threshold [154, 155]. For convenience, we compile also in Table 2 the derived values of the \(\overline{\mathrm{MS}}\) running parameters evaluated at the scale \(\mu _t=m_t^\mathrm{pole}\) following the above prescription.

Table 2 Summary of the input parameters used throughout this work. For convenience, the derived values of the \(\overline{\mathrm{MS}}\) running parameters evaluated at the scale \(\mu _t=m_t^\mathrm{pole}\) are also given. In the Wolfenstein parametrization [152] of the CKM matrix, the four independent Wolfenstein parameters can be chosen as \(\lambda \), A, \(\bar{\rho }=\rho (1-\lambda ^2/2)\), and \(\bar{\eta }=\eta (1-\lambda ^2/2)\) [153]

Concerning the needed entries of the CKM matrix, we adopt the Wolfenstein parametrization [152], generalized to include the higher-order terms of \(\lambda =|V_{us}|\) [153]. To be consistent with the study of NP, the four Wolfenstein parameters \(\lambda \), A, \(\bar{\rho }=\rho (1-\lambda ^2/2)\), and \(\bar{\eta }=\eta (1-\lambda ^2/2)\) should be fitted to the tree-level observables as well as the ratio of the mass differences between the neutral B-meson eigenstates, \(\Delta M_d/\Delta M_s\), which are all expected to have only a minimal sensitivity to the charged-Higgs contributions [60]. To this end, we shall use as input the central values of the fitting results obtained in Ref. [75]. For any further details, the readers are referred to Ref. [75].

As pointed out already in Sect. 2, only the parameters \(\varsigma _u\), \(\varsigma _d\), and \(m_{H^\pm }\) are involved for the rare decays \(t \rightarrow cg(g)\) in the A2HDM. Let us now fix the allowed ranges of these model parameters. For simplicity, we shall entertain ourselves with the CP-conserving limit and assume that the observed \(125\,\mathrm{GeV}\) Higgs is the lightest CP-even scalar of the model, which corresponds exactly to the light scenario considered in Ref. [75]. For the two alignment parameters \(\varsigma _{u,d}\), we choose the ranges

$$\begin{aligned} |\varsigma _{u}| \in [0,1.5], \quad |\varsigma _{d}| \in [0,50], \end{aligned}$$
(4.1)

as constrained mostly by the precision flavour physics, such as \(Z\rightarrow b\bar{b}\), \(\bar{B}\rightarrow X_s\gamma \), as well as \(B_{s,d}^0-\bar{B}_{s,d}^0\) and \(K^0-\bar{K}^0\) mixings [60,61,62,63,64]. It is especially interesting to note that, for large values of \(|\varsigma _{u} \varsigma _{d}|\), important correlated constraints on \(\varsigma _{u}\) and \(\varsigma _{d}\) can be derived from the weak radiative decay \(\bar{B}\rightarrow X_s\gamma \), with the region \(\varsigma _{d} \varsigma _{u}<0\) almost excluded by the current data on the branching ratio of \(\bar{B}\rightarrow X_s\gamma \) [60]. The ranges in Eq. (4.1) are also roughly consistent with the perturbativity requirement, \(|\sqrt{2}\varsigma _{u,d} m_{u,d}/v| \lesssim 1\). For the charged-Higgs mass, we focus only on the case with \(m_{H^\pm }\) varied within the ranges \(m_{H^\pm }\in [200, 600]\,\mathrm{GeV}\) for the \(t \rightarrow cg\) and \(m_{H^\pm }\in [200, 400]\,\mathrm{GeV}\) for the \(t \rightarrow cgg\) decay. Here the lower limit is chosen to be larger than the top-quark mass such that the decay mode \(t \rightarrow b H^+\) is kinematically forbidden, while the upper limit is motivated by the observation that the maximum branching ratio of \(t \rightarrow cg(g)\) for \(m_{H^\pm }\gtrsim 600\,(400)\,\mathrm{GeV}\) approaches already the corresponding SM prediction, as will be demonstrated in the next subsection.Footnote 7 Recently, a detailed global fit has been performed to the A2HDM by considering a CP-conserving scalar potential and real alignment parameters, and including the most constraining flavour observables, electroweak precision measurements, and the available data on Higgs signal strengths and collider searches for heavy scalars, together with the theoretical requirements of perturbativity and positivity of the scalar potential [75]. We shall use the relevant constraints on the alignment parameters \(\varsigma _{u,d}\) resulting from such a global fit obtained at the \(95.5\%\) CL (cf. the brown region in the left plot of Fig. 9) in Ref. [75]; for further details, the readers are also referred to Ref. [75].

4.2 \(\mathcal {B}(t \rightarrow cg(g))\) in the A2HDM

Firstly, with the input parameters collected in Table 2, we give for reference the partial width of the dominant tree-level \(t \rightarrow b W^+\) decay,

$$\begin{aligned} \Gamma (t \rightarrow b W^+)=1.35\,(1.47)\,\mathrm{GeV}, \end{aligned}$$
(4.2)

at the NLO (LO) in QCD, which is found to be well consistent with the current data on the top-quark total width, \(\Gamma _\mathrm {tot}^\mathrm {exp}(t) = 1.42^{+0.19}_{-0.15}\,\mathrm{GeV}\) [151]. This in turn implies that the extra contribution from the partial width \(\Gamma (t \rightarrow b H^+)\) in the \(m_{H^{\pm }} < m_t - m_b\) case should be small, supporting therefore our choice of the lower limit of the charged-Higgs mass, \(m_{H^{\pm }} \ge 200\,\mathrm{GeV}\), as required to forbid the decay \(t \rightarrow b H^+\) kinematically. Otherwise, for \(m_{H^{\pm }}<150\,\mathrm{GeV}\), the alignment parameter \(\varsigma _{d}\) is constrained to be small by the direct charged-Higgs searches at the LHC via the decay \(t \rightarrow b H^+\) [157, 159], implying a very strong suppression on the decay rates of \(t \rightarrow c V\) (\(V = \gamma , Z\)) and \(t \rightarrow c h\) processes [12].

To display the dependence of the branching ratios of \(t\rightarrow cg\) and \(t\rightarrow cgg\) decays on the alignment parameters \(\varsigma _{u,d}\), we fix the charged-Higgs mass to three benchmark values, \(m_{H^{\pm }}=200\), 300, and \(500\,(400)\,\mathrm{GeV}\). In this way, we obtain numerically

$$\begin{aligned}&\mathcal {B}^{\text {A2HDM}}(t \rightarrow cg) = 4.50\times 10^{-12} \nonumber \\&\quad \times {\left\{ \begin{array}{ll} \begin{aligned} \Bigl [1 &{}+ 2.46\times 10^{-5}\, \varsigma _u^4 + 7.19\times 10^{-5}\, \varsigma _d \varsigma _u^3 + 0.261\, \varsigma _d^2 \varsigma _u^2 \\ &{} - 1.30\times 10^{-2}\, \varsigma _d^3 \varsigma _u + 1.63\times 10^{-4}\, \varsigma _d^4 + 4.91\times 10^{-6}\, \varsigma _u^2 \\ &{} + 0.848\, \varsigma _d \varsigma _u - 2.12\times 10^{-2}\, \varsigma _d^2\,\Bigr ], \quad \text {for}\, m_{H^\pm }=200\,\mathrm{GeV}, \\ \Bigl [1 &{}+ 1.96\times 10^{-7}\, \varsigma _u^4 + 2.88\times 10^{-6}\, \varsigma _d \varsigma _u^3 + 3.48\times 10^{-2}\, \varsigma _d^2 \varsigma _u^2 \\ &{} - 1.49\times 10^{-3}\, \varsigma _d^3 \varsigma _u + 1.60\times 10^{-5}\, \varsigma _d^4 + 2.92\times 10^{-6}\, \varsigma _u^2 \\ &{} + 0.310\, \varsigma _d \varsigma _u - 6.63\times 10^{-3}\, \varsigma _d^2\,\Bigr ], \quad \text {for} \,m_{H^\pm }=300\,\mathrm{GeV}, \\ \Bigl [1 &{}+ 3.42\times 10^{-9}\, \varsigma _u^4 + 1.49\times 10^{-7}\, \varsigma _d \varsigma _u^3 + 4.64\times 10^{-3}\, \varsigma _d^2 \varsigma _u^2 \\ &{} - 1.73\times 10^{-4}\, \varsigma _d^3 \varsigma _u + 1.61\times 10^{-6}\, \varsigma _d^4 + 5.41\times 10^{-7}\, \varsigma _u^2 \\ &{} + 0.113\, \varsigma _d \varsigma _u - 2.11\times 10^{-3}\, \varsigma _d^2\,\Bigr ], \quad \text {for}\, m_{H^\pm }=500\,\mathrm{GeV}, \end{aligned} \end{array}\right. } \end{aligned}$$
(4.3)

for the two-body decay \(t \rightarrow cg\), and

$$\begin{aligned}&\mathcal {B}^{\text {A2HDM}}(t \rightarrow cgg) = 8.31\times 10^{-10} \nonumber \\&\quad \times {\left\{ \begin{array}{ll} \begin{aligned} \Bigl [1 &{}+ 6.31\times 10^{-7}\, \varsigma _u^4 + 2.47\times 10^{-7}\, \varsigma _d \varsigma _u^3 + 1.06\times 10^{-3}\, \varsigma _d^2 \varsigma _u^2 \\ &{} - 4.67\times 10^{-5}\, \varsigma _d^3 \varsigma _u + 5.67\times 10^{-7}\, \varsigma _d^4 - 1.19\times 10^{-6}\, \varsigma _u^2 \\ &{} + 2.72\times 10^{-3}\, \varsigma _d \varsigma _u - 9.63\times 10^{-5}\, \varsigma _d^2\,\Bigr ], \quad \text {for}\, m_{H^\pm }=200\,\mathrm{GeV}, \\ \Bigl [1 &{}+ 4.99\times 10^{-8}\, \varsigma _u^4 + 1.40\times 10^{-8}\, \varsigma _d \varsigma _u^3 + 1.35\times 10^{-4}\, \varsigma _d^2 \varsigma _u^2 \\ &{} - 5.20\times 10^{-6}\, \varsigma _d^3 \varsigma _u + 5.47\times 10^{-8}\, \varsigma _d^4 - 3.11\times 10^{-7}\, \varsigma _u^2 \\ &{} + 1.02\times 10^{-3}\, \varsigma _d \varsigma _u - 3.00\times 10^{-5}\, \varsigma _d^2\,\Bigr ], \quad \text {for}\, m_{H^\pm }=300\,\mathrm{GeV}, \\ \Bigl [1 &{}+ 1.32\times 10^{-8}\, \varsigma _u^4 + 3.17\times 10^{-9}\, \varsigma _d \varsigma _u^3 + 4.12\times 10^{-5}\, \varsigma _d^2 \varsigma _u^2 \\ &{} - 1.49\times 10^{-6}\, \varsigma _d^3 \varsigma _u + 1.45\times 10^{-8}\, \varsigma _d^4 - 1.52\times 10^{-7}\, \varsigma _u^2 \\ &{} + 5.78\times 10^{-4}\, \varsigma _d \varsigma _u - 1.54\times 10^{-5}\, \varsigma _d^2\,\Bigr ], \quad \text {for}\, m_{H^\pm }=400\,\mathrm{GeV}, \end{aligned} \end{array}\right. } \end{aligned}$$
(4.4)

for the three-body decay \(t \rightarrow cgg\). Here, the SM results are obtained by taking the limits \(\varsigma _{u,d}\rightarrow 0\) and read

$$\begin{aligned} \mathcal {B}^{\text {SM}}(t \rightarrow cg)&= 4.50\times 10^{-12}, \end{aligned}$$
(4.5)
$$\begin{aligned} \mathcal {B}^{\text {SM}}(t \rightarrow cgg)&= 8.31\times 10^{-10}. \end{aligned}$$
(4.6)

It is found that our SM predictions are generally consistent with that made in Refs. [14, 27] for the two-body decay \(t \rightarrow cg\), as well as in Ref. [39] for the three-body decay \(t \rightarrow cgg\). Especially, the branching ratio of the three-body decay \(t \rightarrow cgg\) is predicted to be about two orders of magnitude larger than that of the two-body decay \(t \rightarrow cg\), which is a clear demonstration of the “higher-order dominance” phenomenon observed already in Ref. [39]. It should be noted that the branching ratio of \(t \rightarrow cgg\) depends on the cutoff parameter C, and the results given by Eqs. (4.4) and (4.6) are obtained with the default value \(C=0.01\). The cutoff dependence of this branching ratio within the SM is shown in Table 3 for six different values of C, together with the comparison with that obtained in Ref. [39]. It can be seen that, although being increased by about \(7\%\) compared to that obtained in Ref. [39] for the same choice of C,Footnote 8 our updated results depend only slightly on the cutoff parameter C, with the variation being only about two times when C varies within the range [0.001, 0.05]. From Eqs. (4.3) and (4.4), one can also see that the branching ratios of \(t \rightarrow cg(g)\) decays display a different dependence on the alignment parameters \(\varsigma _{u,d}\): Firstly, due to the much smaller coefficients, together with the allowed ranges of \(\varsigma _{u}\) and \(\varsigma _{d}\) specified by Eq. (4.1), the terms proportional to \(\varsigma _u^4\), \(\varsigma _d \varsigma _u^3\), and \(\varsigma _u^2\) can be safely neglected. Secondly, for moderate values of \(|\varsigma _d|\) and large values of \(|\varsigma _u|\), the branching ratios will be dominated by the product \(\varsigma _{d} \varsigma _{u}\) and, to a less extent, also by \(\varsigma _{d}^2 \varsigma _{u}^2\). Finally, the significance of \(\varsigma _{d}\) occurs only when \(|\varsigma _d|\) and \(|\varsigma _u|\) take much larger and smaller values, respectively.

Table 3 The cutoff dependence of the branching ratio of \(t \rightarrow cgg\) decay within the SM, as well as the comparison with that obtained in Ref. [39]. Note that the \(\overline{\mathrm{MS}}\) running masses of both the top and charm quarks as well as the \(\alpha \) scheme for the electroweak parameters were used in Ref. [39]
Fig. 3
figure 3

The branching ratio \(\mathcal {B}(t\rightarrow cg)\) as a function of the alignment parameters \(\varsigma _u\) (left) and \(\varsigma _d\) (right), for three benchmark values of the charged-Higgs mass, \(m_{H^{\pm }}=200\), 300, and \(500\,\mathrm{GeV}\), represented by the orange (top), brown (middle), and dark green (bottom) scatter plots, respectively. Here we have used the relevant constraints on the alignment parameters \(\varsigma _{u,d}\) from a global fit obtained at the \(95.5\%\) CL in Ref. [75]. The thickness of the shaded regions reflects the corresponding allowed range for \(\varsigma _{d(u)}\) for a given value of \(\varsigma _{u(d)}\), as can be read from the brown region of the left plot of Figure 9 in Ref. [75]

Fig. 4
figure 4

The branching ratio \(\mathcal {B}(t\rightarrow cgg)\) as a function of the alignment parameters \(\varsigma _u\) (left) and \(\varsigma _d\) (right), for three benchmark values of the charged-Higgs mass, \(m_{H^{\pm }}=200\), 300, and \(400\,\mathrm{GeV}\), represented by the orange (top), brown (middle), and purple (bottom) scatter plots, respectively. The other captions are the same as in Fig. 3

Picking up randomly the values of the alignment parameters \(\varsigma _{u,d}\) within the brown region (corresponding to the global fit obtained at the \(95.5\%\) CL) shown by the left plot of Figure 9 in Ref. [75], we show in Figs. 3 and 4 the branching ratios \(\mathcal {B}(t\rightarrow cg)\) and \(\mathcal {B}(t\rightarrow cgg)\) as a function of \(\varsigma _u\) (left) and \(\varsigma _d\) (right), for three benchmark values of the charged-Higgs mass, \(m_{H^{\pm }}=200\), 300, and \(500\,(400)\,\mathrm{GeV}\), represented by the orange (top), brown (middle), and dark green (purple) (bottom) scatter plots, respectively. It can be seen that the maximum branching ratios of both \(t \rightarrow cg\) and \(t \rightarrow cgg\) decays are reached in the limits \(|\varsigma _u|\rightarrow 0\) and \(|\varsigma _d|\rightarrow 48\), which correspond to the minimum \(|\varsigma _u|\) and maximum \(|\varsigma _d|\) allowed by the combined constraints from the most constraining observables and the theoretical requirements [75]. Furthermore, almost irrespective of the values of the charged-Higgs mass, the maxima of the branching ratios are always reached with the same set of alignment parameters, \((\varsigma _u,\varsigma _d) \simeq (-0.047,-48)\), when \(m_{H^\pm }\) varies within the ranges \([200, 600]\,\mathrm{GeV}\) for the \(t \rightarrow cg\) and \([200, 400]\,\mathrm{GeV}\) for the \(t \rightarrow cgg\) decay. Numerically, we obtain

$$\begin{aligned} \mathcal {B}^{\text {A2HDM}}_\text {max}(t \rightarrow cg)&= {\left\{ \begin{array}{ll} 3.36\times 10^{-9}, &{} \text {for}\, m_{H^\pm }=200\,\mathrm{GeV},\\ 2.84\times 10^{-10}, &{} \text {for}\, m_{H^\pm }=300\,\mathrm{GeV},\\ 1.82\times 10^{-11}, &{} \text {for}\, m_{H^\pm }=500\,\mathrm{GeV}, \end{array}\right. } \end{aligned}$$
(4.7)

and

$$\begin{aligned} \mathcal {B}^{\text {A2HDM}}_\text {max}(t \rightarrow cgg)&= {\left\{ \begin{array}{ll} 2.95\times 10^{-9}, &{} \text {for} \,m_{H^\pm }=200\,\mathrm{GeV},\\ 9.94\times 10^{-10}, &{} \text {for}\, m_{H^\pm }=300\,\mathrm{GeV},\\ 8.60\times 10^{-10}, &{} \text {for}\, m_{H^\pm }=400\,\mathrm{GeV}. \end{array}\right. } \end{aligned}$$
(4.8)

It can be seen that, with the \(95.5\%\)-CL constraints on the alignment parameters from the global fit taken into account [75], the maximum branching ratios of \(t \rightarrow cg\) and \(t \rightarrow cgg\) decays are both reached at \(m_{H^\pm }=200\,\mathrm{GeV}\) and given by \(3.36\times 10^{-9}\) and \(2.95\times 10^{-9}\) respectively, being therefore of the same order, in the A2HDM. This is obviously different from the SM case, where the predicted branching ratio of \(t \rightarrow cgg\) is about two orders of magnitude larger than that of \(t \rightarrow cg\) (cf. Eqs. (4.5) and (4.6)) [39]. For convenience, we show in Fig. 5 the dependence of the maximum branching ratios of \(t\rightarrow cg\) (left) and \(t\rightarrow cgg\) (right) on the charged-Higgs mass \(m_{H^{\pm }}\), with the alignment parameters fixed at \((\varsigma _u,\varsigma _d) \simeq (-0.047,-48)\).Footnote 9 One can see that the maximum branching ratios of \(t \rightarrow cg\) and \(t \rightarrow cgg\) decays decrease sharply as the charged-Higgs mass \(m_{H^{\pm }}\) increases, and approach the corresponding SM predictions for \(m_{H^\pm }\gtrsim 600\,\mathrm{GeV}\) and \(m_{H^\pm }\gtrsim 400\,\mathrm{GeV}\), respectively.Footnote 10 In addition, the maximum branching ratio \(\mathcal {B}^{\text {A2HDM}}_\text {max}(t \rightarrow cg(g))\) will change only slightly when \(m_{H^\pm }\gtrsim 600~(400)\,\mathrm{GeV}\). These observations motivate our choice of the upper limits for the charged-Higgs mass mentioned before. On the other hand, although being enhanced by about three orders of magnitude relative to the SM prediction, the maximum branching ratio of \(t\rightarrow cg\) decay that can be reached in the A2HDM, \(\mathcal {B}^{\text {A2HDM}}_\text {max}(t \rightarrow cg)=3.36\times 10^{-9}\), is still at least one order of magnitude lower than the projected \(95\%\)-CL limits at the future colliders, which are estimated to be \(3.21 \times 10^{-5}\) at the HL-LHC with 3-\(\text {ab}^{-1}\) luminosity data [107, 108], and of \(\mathcal {O}(10^{-7}\)\(10^{-8})\) at the 100-\(\mathrm {TeV}\) FCC-hh with an integrated luminosity of \(10~\text {ab}^{-1}\) [113, 114], respectively. As a consequence, the predicted branching ratios of \(t \rightarrow cg\) and \(t \rightarrow cgg\) decays in the A2HDM are still out of the expected sensitivities of the future HL-LHC and FCC-hh.

Fig. 5
figure 5

The dependence of the maximum branching ratios of \(t\rightarrow cg\) (left) and \(t\rightarrow cgg\) (right) on the charged-Higgs mass \(m_{H^{\pm }}\), with the alignment parameters fixed at \((\varsigma _u,\varsigma _d) \simeq (-0.047,-48)\). The horizontal dash-dotted lines represent the SM predictions given by Eqs. (4.5) and (4.6), respectively

4.3 \(\mathcal {B}(t \rightarrow cg(g))\) in the conventional 2HDMs with \(\mathcal {Z}_2\) symmetries

Let us now discuss the branching ratios of \(t \rightarrow cg\) and \(t \rightarrow cgg\) decays in the four conventional 2HDMs with \(\mathcal {Z}_2\) symmetries, the Yukawa types of which have been given explicitly in Table 1. As only the charged-Higgs interactions with the quark sector are involved, we are actually left with only two different cases, type-I (type-X) 2HDM with \(\varsigma _d=\varsigma _u=1/\tan {\beta }\), and type-II (type-Y) 2HDM with \(\varsigma _d=-1/\varsigma _u=-\tan {\beta }\). Here, to predict the branching ratios of \(t \rightarrow cg\) and \(t \rightarrow cgg\) decays, we need only the knowledge of the charged-Higgs mass \(m_{H^{\pm }}\) and the parameter \(\tan {\beta }\), both of which receive severe constraints from many flavour processes and theoretical considerations [161,162,163,164]. For convenience, we show in Fig. 6 the dependence of the two branching ratios \(\mathcal {B}(t \rightarrow cg)\) (left) and \(\mathcal {B}(t \rightarrow cgg)\) (right) on \(\tan {\beta }\), for three different choices of \(m_{H^{\pm }}\), in the four conventional 2HDMs with \(\mathcal {Z}_2\) symmetries. It should be noted that the branching ratios of \(t\rightarrow cg\) and \(t\rightarrow cgg\) decays are calculated in these four models by varying \(\tan {\beta }=[0.1,60]\), while the ranges where the resulting branching ratios are not significantly enhanced compared with the corresponding SM predictions are not shown in Fig. 6. It can be seen that the sensitivity of the two branching ratios to \(m_{H^{\pm }}\) occurs at low and high \(\tan \beta \) for the type-I (type-X) and the type-II (type-Y) 2HDM respectively, and such a sensitivity becomes, however, less significant for larger \(m_{H^{\pm }}\).

Fig. 6
figure 6

The branching ratios of \(t\rightarrow cg\) (left) and \(t\rightarrow cgg\) (right) as a function of the parameter \(\tan {\beta }=[0.1,60]\), for three different choices of the charged-Higgs mass \(m_{H^{\pm }}\), in the four conventional 2HDMs with \(\mathcal {Z}_2\) symmetries. The horizontal dash-dotted lines represent the SM predictions. We show only the ranges of \(\tan {\beta }\) where the resulting branching ratios display significant enhancements compared with the corresponding SM predictions. Note that the charged-Higgs mass \(m_{H^{\pm }}\) in type-II (type-Y) 2HDM is already bounded to be larger than \(600\,\mathrm{GeV}\) [165,166,167], while the parameter \(\tan \beta \) in type-I (type-X) 2HDM is constrained to be larger than one [162], both of which imply that no significant enhancements of the branching ratios are possible in these conventional 2HDMs

For the type-II 2HDM, the branching ratio of the weak radiative decay \(\bar{B}\rightarrow X_s\gamma \) plays a significant role in setting a lower bound on the charged-Higgs mass \(m_{H^{\pm }}\), with the latest updated results given by \(m_{H^{\pm }} \gtrsim 800\,\mathrm{GeV}\) at \(95\%\) CL [165] and \(m_{H^{\pm }} \gtrsim 790\,(1510)\,\mathrm{GeV}\) at \(2\sigma \,(1\sigma )\) [166]. These bounds are nearly \(\tan \beta \)-independent for \(\tan \beta > 2\) and show a strong dependence on \(\tan \beta \) only for \(\tan \beta < 2\) [167]. Quite recently, a comprehensive study of the allowed parameter space of the type-II 2HDM has been performed by combining the most recent flavour, collider, and electroweak precision observables with theoretical constraints, with the best fit point lying around \(m_{H^{\pm }} \simeq 2\,\mathrm {TeV}\) and \(\tan \beta \simeq 4\) [166]. The experimental lower bound on \(m_{H^{\pm }}\) from direct collider searches is, however, only \(m_{H^{\pm }} \gtrsim 160\,\mathrm{GeV}\) [157], and hence well below the indirect constraints from flavour physics. With these bounds taken into account, it is then found that the branching ratios of both \(t \rightarrow cg\) and \(t \rightarrow cgg\) decays remain almost identical with the corresponding SM predictions, as given by Eqs. (4.5) and (4.6) respectively. The same conclusion can also be applied to the type-Y 2HDM, because its Yukawa type shares the same coupling pattern for the quarks as of the type-II 2HDM [162]. In fact, for these two types of 2HDMs, we have \(\varsigma _d=-1/\varsigma _u=-\tan {\beta }\) and hence \(\varsigma _d \varsigma _u=-1\), while the region \(\varsigma _{d} \varsigma _{u}<0\) is almost excluded by the current data on the branching ratio of the weak radiative decay \(\bar{B}\rightarrow X_s\gamma \) [60]. This also implies that significant enhancements of the branching ratios of \(t \rightarrow cg\) and \(t \rightarrow cgg\) decays relative to the SM predictions are impossible in these two types of 2HDMs.

For the type-I (type-X) 2HDM, on the other hand, a \(95\%\)-CL lower limit on \(m_{H^{\pm }}\) from the branching ratio of \(\bar{B}\rightarrow X_s\gamma \) can be derived for low \(\tan \beta \) only [167]. As an illustration, a bound \(\tan \beta \lesssim 2.5\) can be set when the robust lower charged-Higgs mass limit \(m_{H^{\pm }} \gtrsim 80\,\mathrm{GeV}\) from the combined charged-Higgs searches at LEP [156] is considered. A detailed analysis of the phenomenological status of the charged-Higgs sector in the conventional 2HDMs with \(\mathcal {Z}_2\) symmetries has been performed by combing all the constraints from direct collider searches for charged and neutral Higgs bosons, as well as the most relevant constraints from flavour physics [162]. Using the combined flavour constraints at \(95\%\) CL obtained there, we extract the values \(\tan \beta \simeq 1.8\) for \(m_{H^\pm }=200\,\mathrm{GeV}\), \(\tan \beta \simeq 1.5\) for \(m_{H^\pm }=300\,\mathrm{GeV}\), and \(\tan \beta \simeq 1.2\) for \(m_{H^\pm }=500\,\mathrm{GeV}\) [162]. With these typical values taken as inputs, the corresponding branching ratios of \(t \rightarrow cg\) and \(t \rightarrow cgg\) decays are given in Table 4. It can be seen that, in these two types of 2HDMs, the branching ratio of \(t\rightarrow cg\) decay is predicted to be of the same order as in the SM (cf. Eq. (4.5)), while the branching ratio of \(t\rightarrow cgg\) decay remains almost identical with the SM expectation (cf. Eq. (4.6)).

Table 4 The branching ratios of \(t \rightarrow cg\) and \(t \rightarrow cgg\) decays in the type-I (type-X) 2HDM, with three sets of input parameters of \((m_{H^\pm },\tan \beta )\) extracted from Ref. [162]

In fact, at low \(\tan \beta \), the flavour constraints are similar among all the four conventional 2HDMs with \(\mathcal {Z}_2\) symmetries, because the contributions in this case are now dominated by the universal charged-Higgs couplings to the up-type quarks (cf. Table 1). Then, it is found that values of \(\tan \beta <1\) are ruled out for all values of \(m_{H^\pm } <650\,\mathrm{GeV}\), irrespective of the Yukawa types considered [162]. As a summary, we can therefore conclude that, compared with the SM predictions, no significant enhancements are observed in the four conventional 2HDMs with \(\mathcal {Z}_2\) symmetries for the branching ratios of both \(t \rightarrow cg\) and \(t \rightarrow cgg\) decays, once the flavour constraints on the model parameters are taken into account.

4.4 Further discussions and comments

Let us now explain why the “higher-order dominance” phenomenon is more obvious within the SM than in the A2HDM as well as the four conventional 2HDMs with \(\mathcal {Z}_2\) symmetries. Naively, one would expect that the rate of the two-body decay \(t \rightarrow cg\) is larger than that of the three-body decay \(t \rightarrow cgg\), due to the coupling-constant and phase-space suppression of the latter. This is, however, not true within the SM; instead, we find that the predicted branching ratio of \(t \rightarrow cgg\) is about two orders of magnitude larger than that of \(t \rightarrow cg\) (cf. Eqs. (4.5) and (4.6)) [39]. To understand such a “higher-order dominance” phenomenon, one should firstly note that both the \(t \rightarrow cg\) and \(t \rightarrow cgg\) decays are dominated by the one-loop effective vertex \(V_{tcg^{*}}^\mu \), which can be generically written as [3]

$$\begin{aligned} V_{tcg^{*}}^{\mu }&= \left( k^{2} \gamma ^{\mu } - k^{\mu } k\!\!\!/\right) \left( F_{1}^{L} P_L+F_{1}^{R} P_R\right) \nonumber \\&\quad + i \sigma ^{\mu \nu } k_{\nu }\left( F_{2}^{L} m_{c} P_L+F_{2}^{R} m_{t} P_R\right) , \end{aligned}$$
(4.9)

as required by Lorentz and gauge invariance. Here we have suppressed both the colour labels and a sum over the internal quark flavours together with the CKM matrix elements. The form factors \(F_{1}^{L,R}\) and \(F_{2}^{L,R}\) are a function of the gluon virtuality \(k^2\), and are called the monopolar (or charge radius) and the dipolar (or dipole moment) form factors, respectively. Note that when \(k^2 \rightarrow 0\), i.e., when the gluon goes on-shell, the \(F_{1}^{L,R}\) contributions vanish. Therefore, the two-body decay \(t \rightarrow cg\) is entirely determined by the dipole structure, while both the dipole and the monopole structure contribute to the three-body decay \(t \rightarrow cgg\). It also turns out that the contribution from the monopolar form factors can be considerably larger than from the dipolar ones within the SM [39,40,41,42]. This explains the “higher-order dominance” phenomenon observed within the SM. For the A2HDM, however, the charged-Higgs enhancements are mostly through the dipolar form factors, and hence show up only through a greatly enhanced branching ratio of the two-body decay \(t \rightarrow cg\). This helps us to understand why the branching ratios of \(t \rightarrow cg\) and \(t \rightarrow cgg\) decays are predicted to be of the same order in the A2HDM. For the four conventional 2HDMs with \(\mathcal {Z}_2\) symmetries, on the other hand, after taking into account the relevant flavour constraints on the model parameters, significant enhancements are no longer allowed for the branching ratios of both \(t \rightarrow cg\) and \(t \rightarrow cgg\) decays.

Finally, we would like to make a brief comment on the observability of \(t \rightarrow cgg\) decay. Note that this process can generally be treated in twofold way: One can either treat it inclusively with \(t \rightarrow cg\) or consider it as a separate channel [21]. The former means that the process \(t \rightarrow cgg\) can be considered as a QCD correction to the \(t \rightarrow cg\) decay by assuring that two of the three final-state jets are collinear so that only two can be resolved by the detector. The latter is especially promising if the branching ratio of \(t \rightarrow cgg\) is considerably larger than that of \(t \rightarrow cg\), as observed within the SM [39]. However, to detect \(t \rightarrow cgg\) before \(t \rightarrow cg\), one must avoid collinearity by applying certain cuts. To this end, the cutoff parameter C introduced in the phase space plays an essential role in distinguishing the \(t \rightarrow cgg\) from the \(t \rightarrow cg\) decay. Here, C must be taken in the range of jet-energy resolution of the detector, and a better jet resolution will be definitely welcome.

5 Conclusion

In this paper, we have firstly updated the SM predictions for the branching ratios of both \(t \rightarrow cg\) and \(t \rightarrow cgg\) decays, with the results given respectively by \(\mathcal {B}^{\text {SM}}(t \rightarrow cg) = 4.50\times 10^{-12}\) and \(\mathcal {B}^{\text {SM}}(t \rightarrow cgg) = 8.31\times 10^{-10}\), finding that the latter is about two orders of magnitude larger than the former, which is a clear demonstration of the “higher-order dominance” phenomenon observed in Ref. [39]. This is due to the fact that the contribution from the monopolar form factors can be considerably larger than from the dipolar ones within the SM [39,40,41,42]. Note that the branching ratio of the three-body decay \(t \rightarrow cgg\) is obtained with the cutoff parameter \(C=0.01\).

To check if such a “higher-order dominance” phenomenon is also applied in the A2HDM as well as in the four conventional 2HDMs with \(\mathcal {Z}_2\) symmetries, we have also evaluated the maximum branching ratios of \(t \rightarrow cg\) and \(t \rightarrow cgg\) decays that can be reached in these different models. After taking into account the relevant constraints on the model parameters resulting from a global fit obtained at the \(95.5\%\) CL [75], we find that the branching ratios of \(t \rightarrow cg\) and \(t \rightarrow cgg\) decays can reach up to \(3.36\times 10^{-9}\) and \(2.95\times 10^{-9}\) respectively, being therefore of the same order, in the A2HDM. This is obviously different from the SM case, and can be understood by the fact that the charged-Higgs enhancements are mostly through the dipolar form factors, and hence show up only through a greatly enhanced branching ratio of the two-body decay \(t \rightarrow cg\). On the other hand, compared with the SM predictions, significant enhancements are no longer possible in the four conventional 2HDMs with \(\mathcal {Z}_2\) symmetries for the branching ratios of these two decays, once the flavour constraints on the model parameters [162, 166] are properly taken into account.

Nevertheless, the predicted branching ratios of \(t \rightarrow cg\) and \(t \rightarrow cgg\) decays in the A2HDM are still out of the expected sensitivities of the future HL-LHC and FCC-hh. To have an opportunity to detect \(t \rightarrow cgg\) before \(t \rightarrow cg\), we have to improve the jet resolution of the future detector. Note that, due to the imaginary components of the loop functions, the CKM matrix elements, as well as the complex alignment parameters \(\varsigma _{u,d}\) in the A2HDM, it would also be interesting to investigate the CP asymmetries in these rare top-quark decays [14, 27].