Top quark decays with flavor violation in the B-LSSM

Abstract

The decays of top quark $t\rightarrow c\gamma ,\;t\rightarrow cg,\;t\rightarrow cZ,\;t\rightarrow ch$ are extremely rare processes in the standard model (SM). The predictions on the corresponding branching ratios in the SM are too small to be detected in the future, hence any measurable signal for the processes at the LHC is a smoking gun for new physics. In the extension of minimal supersymmetric standard model with an additional local $U(1)_B\text {--}L$ gauge symmetry (B-LSSM), new gauge interaction and new flavor changing interaction affect the theoretical evaluations on corresponding branching ratios of those processes. In this work, we analyze those processes in the B-LSSM, under a minimal flavor violating assumption for the soft breaking terms. Considering the constraints from updated experimental data, the numerical results imply $Br(t\rightarrow c\gamma )\sim 5\times 10^{-7}$, $Br(t\rightarrow cg)\sim 2\times 10^{-6}$, $Br(t\rightarrow cZ)\sim 4\times 10^{-7}$ and $Br(t\rightarrow ch)\sim 3\times 10^{-9}$ in our chosen parameter space. Simultaneously, new gauge coupling constants $g_{_B},\;g_{_{YB}}$ in the B-LSSM can also affect the numerical results of $Br(t\rightarrow c\gamma ,\;cg,\;cZ,\;ch)$.

1 Introduction

Since the running LHC provides an opportunity to seek out top quark rare decays, top quark shows great promise in revealing the secret of new physics beyond the standard model (SM). The branching ratios of the flavor changing neutral current (FCNC) of top quark $t\rightarrow c\gamma$, $t\rightarrow cg$, $t\rightarrow ch$, $t\rightarrow cZ$ in the SM are highly suppressed [1,2,3,4,5,6,7,8]

$$\begin{aligned}&\mathrm{Br}(t\rightarrow c\gamma )\sim 5\times 10^{-14},\quad \; \mathrm{Br}(t\rightarrow cg)\sim 5\times 10^{-12}, \nonumber \\&\mathrm{Br}(t\rightarrow ch)\sim 3\times 10^{-15},\quad \; \mathrm{Br}(t\rightarrow cZ)\sim 1\times 10^{-14}, \end{aligned}$$

(1)

and beyond the detection capabilities of LHC in the near future. Nevertheless the exotic mechanism from the new physics can enhance those branching ratios drastically [8], which can be detected in the future. The updated upper limits on the branching ratios of LHC are [9,10,11,12,13,14,15,16,17]

$$\begin{aligned}&\mathrm{Br}(t\rightarrow c\gamma )<1.7\times 10^{-3},\quad \; \mathrm{Br}(t\rightarrow cg)<2\times 10^{-4}, \nonumber \\&\mathrm{Br}(t\rightarrow ch)<2.2\times 10^{-3},\quad \; \mathrm{Br}(t\rightarrow cZ)<2.3\times 10^{-4}. \end{aligned}$$

(2)

Hence, detecting those rare top quark decays on the LHC provides a good window to search the new physics beyond the SM. Actually several extensions of the SM predict the branching ratios of the rare top decays surpassing the SM evaluations several orders numerically. In Table 1, we present the theoretical predictions on the branching ratios of those rare decays of top quark in some popular new physics extensions of the SM. Those new physics models include the two-Higgs doublet models with flavour-conservation (FC 2HDM) [2, 3, 18, 19] and without flavour-conservation (NFC 2HDM) [20], minimal supersymmetric extension of the SM (MSSM) [21,22,23], supersymmetry (SUSY) without R-parity [24, 25], the Topcolour-assisted Technicolour model (TC2) [26] and the extension with warped extra dimensions (RS) [27]. The results in Refs. [21,22,23] were obtained when the supersymmetric particles were not constrained strongly by direct searches at the LHC. In addition, the results in Refs. [21, 22] under a minimal flavor violating assumption for the soft breaking terms, while Ref. [23] takes into account the off-diagonal terms for the squark matrices. Hence the results in Ref. [23] are larger than those in Refs. [21, 22].

Table 1 FCNC decays branching ratios of several SM extension

In the supersymmetric extensions of the SM, the extension with local B–L gauge symmetry (here employing the abbreviation B-LSSM denoting the extension) draw the attention of physicists, since it provides the candidate for the cold dark matter and the seesaw mechanism to explain the oscillation of neutrinos naturally. The model B-LSSM [28, 29] is based on the gauge symmetry group $SU(3)_C\otimes SU(2)_L\otimes U(1)_Y\otimes U(1)_{B\text {--}L}$, where B stands for the baryon number and L stands for the lepton number respectively. Besides accounting elegantly for the existence and smallness of the left-handed neutrino masses, the B-LSSM also alleviates the aforementioned little hierarchy problem of the MSSM [30], because the exotic singlet Higgs and right-handed (s)neutrinos [31,32,33,34,35,36,37] release additional parameter space from the LEP, Tevatron and LHC constraints.

In this model, the invariance under $U(1)_{B\text {--}L}$ gauge group imposes the the R-parity conservation which is assumed in the MSSM to avoid proton decay. And R-parity conservation can be maintained if $U(1)_{B\text {--}L}$ symmetry is broken spontaneously [38]. Furthermore, it could help to understand the origin of R-parity and its possible spontaneous violation in the supersymmetric models [39,40,41] as well as the mechanism of leptogenesis [42, 43]. Moreover, the model can provide much more candidates for the Dark Matter comparing that with the MSSM [44,45,46,47].

In this work, we analyze those processes in the B-LSSM, under a minimal flavor violating assumption for the soft breaking terms. In this case, the only source of flavor violation comes from the Cabibbo–Kobayashi–Maskawa (CKM) matrix in the quark sector. And we can explore the effects of new parameters to those processes, with respect to the MSSM. Our presentation is organized as follows. In Sect. 2, the main ingredients of B-LSSM are summarized briefly by introducing the superpotential, the general soft breaking terms and the Higgs sector. In Sect. 3, the branching ratios for $t\rightarrow c\gamma , t\rightarrow cg, t\rightarrow ch$ and $t\rightarrow cZ$ is calculated in the model. The numerical analyses are given in Sect. 4, and Sect. 5 gives a summary.

2 The B-LSSM

In the B-LSSM, one enlarges the local gauge group of the SM to $SU(3)_C\otimes SU(2)_L\otimes U(1)_Y\otimes U(1)_{B\text {--}L}$, where the $U(1)_{B\text {--}L}$ is spontaneously broken by the chiral singlet superfields $\hat{\eta }_1$ and $\hat{\eta }_2$. In literatures there are several popular versions of B-LSSM. Here we adopt the version described in Refs. [48,49,50,51,52] to proceed our analysis, while this version of B-LSSM is encoded in SARAH [53,54,55,56,57] which is used to create the mass matrices and interaction vertexes in the B-LSSM. Besides the superfields of the MSSM, the exotic superfields of the B-LSSM are three generations right-handed neutrinos $\hat{\nu }_i^c\sim$ (1, 1, 0, 1) and two chiral singlet superfields $\hat{\eta }_{1}\sim (1,1,0,-1)$, $\hat{\eta }_{2}\sim (1,1,0,1)$. Meanwhile, quantum numbers of the matter chiral superfields for quarks and leptons are given by

$$\begin{aligned}&\hat{Q}_i\sim (3, 2, 1/6, 1/6), \quad \;\hat{L}_i\sim (1, 2, -1/2, -1/2),\nonumber \\&\;\hat{U}_i\sim (3, 1, -2/3, -1/6),\nonumber \\&\hat{D}_i\sim (3, 1, 1/3, -1/6), \quad \;\hat{E}_i\sim (1, 1, 1, 1/2) \end{aligned}$$

(3)

with $i=1,2,3$ denoting the index of generation. In addition, the quantum numbers of two Higgs doublets is assigned as

$$\begin{aligned}&\hat{H_1}=\left( \begin{array}{c}H_1^1 \\ H_1^2\end{array}\right) \sim (1, 2, -1/2, 0),\nonumber \\&\hat{H_2}=\left( \begin{array}{c}H_2^1 \\ H_2^2\end{array}\right) \sim (1, 2, 1/2, 0). \end{aligned}$$

(4)

The corresponding superpotential of the B-LSSM is written as

$$\begin{aligned}&W=W_{MSSM}+W_{(B-L)}. \end{aligned}$$

(5)

Here, $W_{MSSM}$ is the superpotential of the MSSM, and $W_{(B-L)}$ is the sector involving exotic superfields, and

$$\begin{aligned}&W_{(B-L)}=Y_{\nu , ij}\hat{L_i}\hat{H_2}\hat{\nu }^c_j-\mu ' \hat{\eta }_1 \hat{\eta }_2 +Y_{x, ij} \hat{\nu }_i^c \hat{\eta }_1 \hat{\nu }_j^c, \end{aligned}$$

(6)

where i, j are generation indices. Correspondingly, the soft breaking terms of the B-LSSM are generally given as

$$\begin{aligned} \mathcal {L}_{soft}= & {} \mathcal {L}_{MSSM}+\Big [-M_{BB^{'}}\tilde{\lambda }_{B^{'}} \tilde{\lambda }_{B} - \frac{1}{2}M_{B^{'}}\tilde{\lambda }_{B^{'}} \tilde{\lambda }_{B^{'}} \nonumber \\&-B_{\mu ^{'}}\tilde{\eta }_1 \tilde{\eta }_2 +T_{\nu }^{ij} H_2 \tilde{\nu }_i^c \tilde{L}_j+T_x^{ij} \tilde{\eta }_1 \tilde{\nu }_i^c \tilde{\nu }_j^c \nonumber \\&+h.c.\Big ]-m_{\tilde{\eta }_1}^2 |\tilde{\eta }_1|^2-m_{\tilde{\eta }_2}^2 |\tilde{\eta }_2|^2-m_{\tilde{\nu },ij}^2(\tilde{\nu }_i^c)^* \tilde{\nu }_j^c, \end{aligned}$$

(7)

with $\lambda _{B}, \lambda _{B^{'}}$ denoting the gaugino of $U(1)_Y$ and $U(1)_{(B-L)}$ respectively, $\mathcal {L}_{MSSM}$ is the soft breaking terms in MSSM. The local gauge symmetry $SU(2)_L\otimes U(1)_Y\otimes U(1)_{B\text {--}L}$ breaks down to the electromagnetic symmetry $U(1)_{em}$ as the Higgs fields receive vacuum expectation values (VEVs):

$$\begin{aligned} H_1^1= & {} \frac{1}{\sqrt{2}}(v_1+\mathrm{Re}H_1^1+i\mathrm{Im}H_1^1),\nonumber \\ H_2^2= & {} \frac{1}{\sqrt{2}}(v_2+\mathrm{Re}H_2^2+i\mathrm{Im}H_2^2),\nonumber \\ \tilde{\eta }_1= & {} \frac{1}{\sqrt{2}}(u_1+\mathrm{Re}\tilde{\eta }_1+i\mathrm{Im}\tilde{\eta }_1),\nonumber \\ \tilde{\eta }_2= & {} \frac{1}{\sqrt{2}}(u_2+i\mathrm{Re}\tilde{\eta }_2+i\mathrm{Im}\tilde{\eta }_2)\;. \end{aligned}$$

(8)

For convenience, we define $u^2=u_1^2+u_2^2,\; v^2=v_1^2+v_2^2$ and $\tan \beta ^{'}=\frac{u_2}{u_1}$ in analogy to the ratio of the MSSM VEVs ($\tan \beta =\frac{v_2}{v_1}$).

The presence of two Abelian groups gives rise to a new effect absent in the MSSM or other SUSY models with just one Abelian gauge group: the gauge kinetic mixing. It results from the invariance principle allows the Lagrangian to include a mixing term between the strength tensors of gauge fields associated with the U(1) gauge groups, $-\kappa _{_{Y,BL}}A_{_\mu }^{\prime Y}A^{\prime \mu , BL}$, where $A_{_\mu }^{\prime Y}, A^{\prime \mu , BL}$ denote the gauge fields associated with the two U(1) gauge groups, $Y, B\text {--}L$ corresponding to the hypercharge and B-L charge respectively, $\kappa _{_{Y,BL}}$ is an antisymmetric tensor which includes the mixing of $U(1)_Y$ and $U(1)_{B\text {--}L}$ gauge fields. This mixing couples the B–L sector to the MSSM sector, and even if it is set to zero at $M_{GUT}$, it can be induced through RGEs [58,59,60,61,62,63,64]. In practice, it turns out that it is easier to work with non-canonical covariant derivatives instead of off-diagonal field-strength tensors. However, both approaches are equivalent [65]. Hence in the following, we consider covariant derivatives of the form

$$\begin{aligned}&D_\mu =\partial _\mu -i\left( \begin{array}{cc}Y,&B-L\end{array}\right) \left( \begin{array}{cc}g_{_Y},&{}g_{_{YB}}^{'}\\ g_{_{BY}}^{'},&{}g_{_{B-L}}\end{array}\right) \left( \begin{array}{c}A_{_\mu }^{\prime Y} \\ A_{_\mu }^{\prime BL}\end{array}\right) . \end{aligned}$$

(9)

As long as the two Abelian gauge groups are unbroken, we still have the freedom to perform a change of the basis

$$\begin{aligned}&D_\mu =\partial _\mu -i\left( \begin{array}{cc}Y,&B-L\end{array}\right) \left( \begin{array}{cc}g_{_Y},&{}g_{_{YB}}^{'}\\ g_{_{BY}}^{'},&{}g_{_{B-L}}\end{array}\right) R^TR \left( \begin{array}{c}A_{_\mu }^{\prime Y} \\ A_{_\mu }^{\prime BL}\end{array}\right) , \end{aligned}$$

(10)

where R is a $2\times 2$ orthogonal matrix. Choosing R in a proper form, one can write the coupling matrix as

$$\begin{aligned}&\left( \begin{array}{cc}g_{_Y},&{}g_{_{YB}}^{'}\\ g_{_{BY}}^{'},&{}g_{_{B-L}}\end{array}\right) R^T=\left( \begin{array}{cc}g_{_1},&{}g_{_{YB}}\\ 0,&{}g_{_{B}}\end{array}\right) \;, \end{aligned}$$

(11)

where $g_{_{1}}$ corresponds to the measured hypercharge coupling which is modified in B-LSSM as given along with $g_{_{B}}$ and $g_{_{YB}}$ in [66, 67]. Then, we can redefine the U(1) gauge fields

$$\begin{aligned}&R\left( \begin{array}{c} A_{_\mu }^{\prime Y} \\ A_{_\mu }^{\prime BL}\end{array}\right) = \left( \begin{array}{c}A_{_\mu }^{Y} \\ A_{_\mu }^{BL}\end{array}\right) . \end{aligned}$$

(12)

Immediate interesting consequence of the gauge kinetic mixing arise in various sectors of the model as discused in the subsequent analysis. Firstly, $A^{BL}$ boson mixes at the tree level with the $A^Y$ and $V^3$ bosons. In the basis $(A^Y, V^3, A^{BL})$, the correponding mass matrix reads,

$$\begin{aligned}&\left( \begin{array}{*{20}{c}} \frac{1}{8}g_{_1}^2 v^2 &{} -\frac{1}{8}g_{_1}g_{_2} v^2 &{} \frac{1}{8}g_{_1}g_{_{YB}} v^2 \\ -\frac{1}{8}g_{_1}g_{_2} v^2 &{} \frac{1}{8}g_{_2}^2 v^2 &{} -\frac{1}{8}g_{_2}g_{_{YB}} v^2\\ \frac{1}{8}g_{_1}g_{_{YB}} v^2 &{} -\frac{1}{8}g_{_2}g_{_{YB}} v^2 &{} \frac{1}{8}g_{_{YB}}^2 v^2+\frac{1}{8}g_{_{B}}^2 u^2 \end{array}\right) . \end{aligned}$$

(13)

This mass matrix can be diagonalized by a unitary mixing matrix, which can be expressed by two mixing angles $\theta _{_W}$ and $\theta _{_W}'$ as

$$\begin{aligned}&\left( \begin{array}{*{20}{c}} \gamma \\ Z\\ Z' \end{array}\right) = \left( \begin{array}{*{20}{c}} \cos \theta _{_W} &{} \sin \theta _{_W} &{} 0 \\ -\sin \theta _{_W}\cos \theta _{_W}' &{} \cos \theta _{_W}\cos \theta _{_W}' &{} \sin \theta _{_W}'\\ \sin \theta _{_W}\sin \theta _{_W}' &{} -\cos \theta _{_W}'\sin \theta _{_W}' &{} \cos \theta _{_W}' \end{array}\right) \left( \begin{array}{*{20}{c}} A^Y\\ V^3\\ A^{BL} \end{array}\right) .\nonumber \\ \end{aligned}$$

(14)

Then $\sin ^2\theta _{_W}'$ can be written as

$$\begin{aligned}&\sin ^2\theta _{_W}'=\frac{1}{2}\nonumber \\&\quad -\frac{(g_{_{YB}}^2-g_{_1}^2-g_{_2}^2)x^2+ 4g_{_B}^2}{2\sqrt{(g_{_{YB}}^2+g_{_1}^2+g_{_2}^2)x^4+8g_{_B}^2(g_{_{YB}}^2-g_{_1}^2-g_{_2}^2x^2)+16g_{_B}^2}},\nonumber \\ \end{aligned}$$

(15)

where $x=\frac{v}{u}$. Compared with the MSSM, this $Z-Z'$ mixing makes new contributions to the $t\rightarrow cZ$ decay channel, and the order of magnitude of $\sin \theta _{_W}'$ is about $\mathcal {O}(10^{-2})$ [68, 69]. The exact eigenvalues of Eq.(13) are given by

$$\begin{aligned} \;m_\gamma ^2= & {} 0,\nonumber \\ \;m_{Z,{Z^{'}}}^2= & {} \frac{1}{8}\Big ((g_{_1}^2+g_2^2+g_{_{YB}}^2)v^2+4g_{_B}^2u^2 \nonumber \\&\mp \sqrt{(g_{_1}^2+g_{_2}^2+g_{_{YB}}^2)^2v^4+8(g_{_{YB}}^2-g_{_1}^2- g_{_2}^2)g_{_B}^2v^2u^2+16g_{_B}^4u^4}\Big ),\nonumber \\ \end{aligned}$$

(16)

In addition, the charged Higgs boson and W gauge boson mass can be written as

$$\begin{aligned} \;m_{H^\pm }^2= & {} \frac{4B_\mu (1+\tan \beta ^2)+g_2^2 v_1^2\tan \beta (1+\tan \beta ^2)}{4\tan \beta },\nonumber \\ \;m_W^2= & {} \frac{1}{4}g_{_2}^2v^2. \end{aligned}$$

(17)

Then the gauge kinetic mixing leads to the mixing between the $H_1^1,\;H_2^2,\;\tilde{\eta _1},\;\tilde{\eta _2}$ at the tree level. In the basis ($\mathrm{Re}H_1^1$, $\mathrm{Re}H_2^2$, $\mathrm{Re}\tilde{\eta }_1$, $\mathrm{Re}\tilde{\eta }_2$), the tree level mass squared matrix for scalar Higgs bosons is given by

$$\begin{aligned} M_h^2= & {} u^2\nonumber \\&\times \left( \begin{array}{*{20}{c}} {\frac{1}{4}\frac{g^2 x^2}{1+\tan \beta ^2}+n^2\tan \beta }&{}{-\frac{1}{4}g^2\frac{x^2\tan \beta }{1+\tan ^2\beta }}-n^2&{} {\frac{1}{2}g_{_B}g_{_{YB}}\frac{x}{T}}&{} {-\frac{1}{2}g_{_B}g_{_{YB}}\frac{x\tan \beta '}{T}}\\ {-\frac{1}{4}g^2\frac{ x^2\tan \beta }{1+\tan ^2\beta }}-n^2&{}{\frac{1}{4}\frac{g^2\tan ^2\beta x^2}{1+\tan \beta ^2}+\frac{n^2}{\tan \beta }}&{} {\frac{1}{2}g_{_B}g_{_{YB}}\frac{x\tan \beta }{T}}&{}{\frac{1}{2}g_{_B}g_{_{YB}}\frac{x\tan \beta \tan \beta '}{T}}\\ {\frac{1}{2}g_{_B}g_{_{YB}}\frac{x}{T}}&{}{\frac{1}{2}g_{_B}g_{_{YB}}\frac{x\tan \beta }{T}}&{}{\frac{g_{_B}^2}{1+\tan ^2\beta '}+\tan \beta 'N^2}&{} {-g_{_B}^2\frac{\tan \beta '}{1+\tan ^2\beta '}-N^2}\\ {-\frac{1}{2}g_{_B}g_{_{YB}}\frac{x\tan \beta '}{T}}&{}{\frac{1}{2}g_{_B}g_{_{YB}}\frac{x\tan \beta \tan \beta '}{T}}&{} {-g_{_B}^2\frac{\tan \beta '}{1+\tan ^2\beta ^{'}}-N^2}&{}{g_{_B}^2\frac{\tan ^2\beta '}{1+\tan ^2\beta '}+\frac{N^2}{\tan \beta '}} \end{array}\right) \end{aligned}$$

(18)

where $g^2=g_{_1}^2+g_{_2}^2+g_{_{YB}}^2$, $T=\sqrt{1+\tan ^2\beta }\sqrt{1+\tan ^2\beta '}$, $n^2=\frac{\mathrm{Re}B\mu }{u^2}$ and $N^2=\frac{\mathrm{Re}B\mu ^{'}}{u^2}$, respectively. Compared the MSSM, this new mixing in the B-LSSM can affect the theoretical prediction of the process $t\rightarrow ch$.

Including the leading-log radiative corrections from stop and top quark, the mass of the SM-like Higgs boson can be written as [70,71,72]

$$\begin{aligned} \Delta m_h^2= & {} \frac{3m_t^4}{2\pi v^2}\Big [\Big (\tilde{t}+\frac{1}{2}+\tilde{X}_t\Big )\nonumber \\&+\frac{1}{16\pi ^2}\Big (\frac{3m_t^2}{2v^2}-32\pi \alpha _3\Big )\Big (\tilde{t}^2 +\tilde{X}_t \tilde{t}\Big )\Big ],\nonumber \\ \tilde{t}= & {} log\frac{M_S^2}{m_t^2},\qquad \;\tilde{X}_t=\frac{2\tilde{A}_t^2}{M_S^2}\Big (1-\frac{\tilde{A}_t^2}{12M_S^2}\Big ), \end{aligned}$$

(19)

where $\alpha _3$ is the strong coupling constant, $M_S=\sqrt{m_{{\tilde{t}}_1}m_{{\tilde{t}}_2}}$ with $m_{{\tilde{t}}_{1,2}}$ denoteting the stop masses, $\tilde{A}_t=A_t-\mu \cot \beta$ with $A_t=T_{u,33}$ being the trilinear Higgs stop coupling and $\mu$ denoting the Higgsino mass parameter. Then the SM-like Higgs mass can be written as

$$\begin{aligned}&m_h=\sqrt{(m_{h_1}^0)^2+\Delta m_h^2}, \end{aligned}$$

(20)

where $m_{h_1}^0$ denotes the lightest tree-level Higgs mass.

Meanwhile, additional D-terms contribute to the mass matrices of the squarks and sleptons, and down type squarks affect the subsequent analysis. On the basis $({\tilde{d}}_L, {\tilde{d}}_R)$, mass matrix for down type squarks is given by

$$\begin{aligned} m_{{\tilde{d}}}^2= & {} \left( \begin{array}{cc}m_{dL},&{}\frac{1}{\sqrt{2}}(v_1 T_d^\dagger -v_2\mu Y_d^\dagger )\\ \frac{1}{\sqrt{2}}(v_1 T_d-v_2\mu ^* Y_d),&{}m_{dR}\end{array}\right) , \end{aligned}$$

(21)

$$\begin{aligned} m_{dL}= & {} \frac{1}{24}\Big (2g_{_B}(g_{_B}+g_{_{YB}})(u_2^2-u_1^2)\nonumber \\&+(3g_2^2+g_1^2+g_{_{YB}}^2+ g_{_B}g_{_{YB}})(v_2^2-v_1^2)\Big )\nonumber \\&+m_{{\tilde{q}}}^2+\frac{v_1^2}{2}Y_d^\dagger Y_d,\nonumber \\ m_{dR}= & {} \frac{1}{24}\Big (2g_{_B}(g_{_B}-2g_{_{YB}})(u_1^2-u_2^2)+2(g_1^2\nonumber \\&+g_{_{YB}}^2- \frac{1}{2}g_{_B}g_{_{YB}})(v_2^2-v_1^2)\Big )\nonumber \\&+m_{{\tilde{d}}}^2+\frac{v_1^2}{2}Y_d^\dagger Y_d. \end{aligned}$$

(22)

It can be noted that new gauge coupling constants $g_{_B}$ and $g_{_{YB}}$, with respect to the MSSM, affect the masses of down type squarks significantly when u is large.

3 Theoretical calculation on $t\rightarrow c\gamma , cg, cZ$ and ch processes

In this section, we analyze one-loop radiative corrections to the rare decay processes of top quark $t\rightarrow c\gamma , cg, cZ$ and ch in the B-LSSM. Here the contributions from self-energy are attributed to the renormalization of external quarks wave functions. The dominating triangle diagrams contributing to the rare top quark decay processes $t\rightarrow c\gamma , cg, cZ, ch$ are presented in Figs. 1, 2, respectively.

Fig. 1

The dominating Feynman diagrams contributing to $t\rightarrow c\gamma ,\;g,\;Z$ in the B-LSSM

In the B-LSSM, the corresponding amplitude for the rare decay process $t\rightarrow c\gamma ,\;g,\;Z$ is written as

$$\begin{aligned} \mathcal {M}_{tcV}= & {} (4\pi )^{-2}\varepsilon ^{\mu } \bar{u}_c(p^{'}){\Big (}\kappa _{V L1}\gamma _\mu P_L +i\kappa _{V L2}\sigma _{\mu \nu }q^\nu P_L\nonumber \\&+(L\rightarrow R){\Big )}u_t(p), \end{aligned}$$

(23)

where V denotes $\gamma ,\;g,\;Z$, $u_t$ and $u_c$ denote the wave functions of top quark and charm quark, p is the momentum of top quark, $p^{'}$ is the momentum of charm quark, q is the momentum of vector boson, and $\varepsilon ^\mu$ denotes the polarization vector of photon, gluon and Z boson. Correspondingly, the coefficients $\kappa _{V Li}$ and $\kappa _{V Ri}\;(i=1,2)$ are originating from those Feynman diagrams in Fig. 1. The picture shows that, compared the MSSM, the new definition of the down type squark mass matrix and the corresponding rotation matrix affect the predictions on the processes $t\rightarrow c\gamma , cg, cZ$ in the B-LSSM. In addition, the $Z-Z'$ mixing in Eq. (13) also makes new contributions to the $t\rightarrow cZ$ decay channel.

Then in order to explain how the calculation of the feynman diagrams in Fig. 1 has been performed, we will take the calculation of Fig. 1(1) below for example. The corresponding amplitude can be written as

(24)

where $A_{{\bar{c}}\tilde{D}_k{{\tilde{\chi }}^+_j}L,R},\;A_{{\bar{\tilde{\chi }}^+_j}\tilde{D}_itL,R},\;B_{V\tilde{D}_i\tilde{D}_k}$ denote the constant parts of the interaction vertex about $\bar{c}\tilde{D}_k{{\tilde{\chi }}^+_j},\;{\bar{\tilde{\chi }}^+_j}\tilde{D}_it,\;V\tilde{D}_i \tilde{D}_k$ respectively, L and R in subscript denote the left-hand part and right-hand part, and all of them can be got through SARAH. Apply the transverse wave condition and Dirac equation, the amplitude can be simplify as

$$\begin{aligned} \mathcal {M}_{tcV}^{(1)}= & {} \frac{1}{16\pi ^2}\varepsilon ^{\mu } \bar{u}_c(p^{'})\Big (\kappa _{V L1}^{(1)}\gamma _\mu P_L +i\kappa _{V L2}^{(1)}\sigma _{\mu \nu }q^\nu P_L\nonumber \\&+(L\rightarrow R)\Big )u_t(p) \end{aligned}$$

(25)

where

$$\begin{aligned} \kappa _{V L1}^{(1)}= & {} -i B_{V\tilde{D}_i\tilde{D}_k}\Big [\Big (m_tm_c(-C_2-C_{22})-2C_{00}\Big )\nonumber \\&\times A_{{\bar{\tilde{\chi }}^+_j}\tilde{D}_itR}A_{{\bar{c}}\tilde{D}_k{{\tilde{\chi }}^+_j}L}+\Big (m_t^2(-C_2-C_{22}\nonumber \\&-C_{12})+m_c^2C_{12}\Big ) A_{{\bar{\tilde{\chi }}^+_j}\tilde{D}_itL} A_{{\bar{c}}\tilde{D}_k{{\tilde{\chi }}^+_j}R}\nonumber \\&-m_{{\tilde{\chi }}^+_j}m_cC_2A_{{\bar{\tilde{\chi }}^+_j}\tilde{D}_itL}A_{{\bar{c}}\tilde{D}_k{{\tilde{\chi }}^+_j}L}\nonumber \\&- m_{{\tilde{\chi }}^+_j}m_tC_2A_{{\bar{\tilde{\chi }}^+_j}\tilde{D}_itR}A_{{\bar{c}}\tilde{D}_k{{\tilde{\chi }}^+_j}R}\Big ], \end{aligned}$$

(26)

$$\begin{aligned} \kappa _{V L2}^{(1)}= & {} -i B_{V\tilde{D}_i\tilde{D}_k}\Big [m_t(-C_2-C_{22}-C_{12})\nonumber \\&\times A_{{\bar{\tilde{\chi }}^+_j}\tilde{D}_itR}A_{{\bar{c}}\tilde{D}_k{{\tilde{\chi }}^+_j}L}\nonumber \\&-m_{{\tilde{\chi }}^+_j}C_2A_{{\bar{\tilde{\chi }}^+_j}\tilde{D}_itL}A_{{\bar{c}}\tilde{D}_k{{\tilde{\chi }}^+_j}L}\nonumber \\&+m_cC_{12}A_{t\tilde{D}_i{{\tilde{\chi }}^+_j}L}A_{c\tilde{D}_k{{\tilde{\chi }}^+_j}R}\Big ], \end{aligned}$$

(27)

$$\begin{aligned} \kappa _{V R1,2}^{(1)}= & {} \kappa _{V L1,2}^{(1)} (L\leftrightarrow R). \end{aligned}$$

(28)

Here $C_0, C_1, C_2, C_{00}, C_{11}, C_{12}, C_{22}$ are Passarino–Veltman scalar functions [73], and the argument above is $[m_V^2,m_c^2,m_t^2, m_{\tilde{D}_i}^2,m_{\tilde{D}_k}^2,m_{{\tilde{\chi }}^+_j}^2]$. The other diagrams corresponding to $t\rightarrow cV$ can be calculated similarly and contribute to the operators $\gamma _\mu P_{L,R}$ and $i\sigma _{\mu \nu }q^\nu P_{L,R}$.

Fig. 2

The Feynman diagrams contributing to $t\rightarrow ch$ in the B-LSSM

In the effective coupling for $h\bar{c}t$, there are only two effective operators:

$$\begin{aligned}&\mathcal {M}_{tch}=(4\pi )^{-2}\bar{u}_c(p^{'})(\kappa _{hL}P_L+\kappa _{hR}P_R)u_t(p)\;, \end{aligned}$$

(29)

the coefficients $\kappa _{hL},\;\kappa _{hR}$ are originating from those Feynman diagrams in Fig. 2:

$$\begin{aligned}&\kappa _{hL}=\sum \limits _{a=1}^{6}\kappa _{hL}^{(a)}, \nonumber \\&\kappa _{hR}=\sum \limits _{a=1}^{6}\kappa _{hR}^{(a)}, \end{aligned}$$

(30)

where the contributions $\kappa _{hL}^{(a)},\;\kappa _{hR}^{(a)}$ depend on the relevant Feynman diagrams in the model. Figure 2 shows that, except for the new contributions from down type squarks, the mixing between the Higgs doublets and the exotic singlets $\tilde{\eta }_{1,2}$ also affects the $t\rightarrow ch$ decay channel. And we will take the calculation of Fig. 2(1) below for example. The amplitude can be given as

(31)

Apply the Dirac equation, the amplitude can be simplify as

$$\begin{aligned}&\mathcal {M}_{tch}^{(1)}=\frac{1}{16\pi ^2}\bar{u}_c(p')\Big (\kappa _{hL}P_L+\kappa _{hR}P_R\Big )u_t(p) \end{aligned}$$

(32)

where

$$\begin{aligned} \kappa _{hL}^{(1)}= & {} -i B_{h\tilde{D}_i\tilde{D}_k}\Big [m_t(C_0+C_1+C_2)A_{\bar{f}\tilde{D}_itR}A_{{\bar{c}}\tilde{D}_k{{\tilde{\chi }}^+_j}L}\nonumber \\&-m_cC_1A_{{\bar{\tilde{\chi }}^+_j}\tilde{D}_itL}A_{{\bar{c}}\tilde{D}_k{{\tilde{\chi }}^+_j}R} \end{aligned}$$

(33)

$$\begin{aligned}&+m_{{\tilde{\chi }}^+_j}C_0A_{{\bar{\tilde{\chi }}^+_j}\tilde{D}_itL}A_{\bar{c}\tilde{D}_k{{\tilde{\chi }}^+_j}L}\Big ], \end{aligned}$$

(34)

$$\begin{aligned} \kappa _{hR}^{(1)}= & {} \kappa _{hL}^{(1)} (L\leftrightarrow R). \end{aligned}$$

(35)

the argument in Passarino–Veltman scalar functions above is $[m_h^2,m_c^2,m_t^2,m_{\tilde{D}_i}^2,m_{\tilde{D}_k}^2,m_{{\tilde{\chi }}^+_j}^2]$. The other diagrams corresponding to $t\rightarrow ch$ can be calculated similarly and contribute to the operators $P_L$ and $P_R$.

Based on Eqs. (23) and (29), the corresponding branching ratios of the rare decay processes of top quark respectively read as

$$\begin{aligned}&Br(t\rightarrow cV)\nonumber \\&\quad =\frac{|\mathcal {M}_{tcV}|^2\sqrt{((m_t+m_V)^2-m_c^2)((m_t-m_V)^2-m_c^2)}}{32\pi m_t^3\Gamma _{total}}, \nonumber \\&Br(t\rightarrow ch)\nonumber \\&\quad =\frac{|\mathcal {M}_{tch}|^2\sqrt{((m_t+m_h)^2-m_c^2)((m_t-m_h)^2-m_c^2)}}{32\pi m_t^3\Gamma _{total}}. \end{aligned}$$

(36)

where $\Gamma _{total}=1.4$ GeV [74] is the total decay width of top quark.

Fig. 3

$Br(t\rightarrow c\gamma )$ (a), $Br(t\rightarrow cg)$ (b), $Br(t\rightarrow cZ)$ (c), $Br(t\rightarrow ch)$ (d) versus $m_{{\tilde{b}}}$ for $\tan \beta =15\;(\mathrm{solid\;line}),\;\tan \beta =25\;(\mathrm{dashed\;line}),\;\tan \beta =35$ (dot-dashed line) are plotted

4 Numerical analyses

In this section, we present the numerical results of $Br(t\rightarrow c\gamma ,\;cg,\;cZ,\;ch)$ with the help of LoopTools and FeynCalc [75, 76]. The relevant SM input parameters are chosen as $m_W=80.385\, \mathrm{GeV},\;m_Z=90.19 \,\mathrm{GeV},\;\alpha _{em}(m_Z)=1/128.9,\;\alpha _s(m_Z)=0.118, \;m_t=173.5\,\mathrm{GeV},\;m_c=1.275 \,\mathrm{GeV}$. Meanwhile the CKM matrix is [74]

$$\begin{aligned}&\left( \begin{array}{*{20}{l}} {0.97417}&{}\quad {0.2248}&{}\quad {4.09\times 10^{-3}}\\ {-0.22}&{}\quad {0.995}&{}\quad {4.05\times 10^{-2}}\\ {8.2\times 10^{-3}}&{}\quad {-4\times 10^{-2}}&{}\quad {1.009} \end{array}\right) . \end{aligned}$$

(37)

The updated experimental data [77] on searching $Z^\prime$ indicates $M_{Z^{'}}\ge 4.05\,\mathrm{TeV}$ at 95% Confidence Level (CL), and Refs. [78, 79] give us an upper bound on the ratio between the $Z^{'}$ mass and its gauge coupling at 99% CL as

$$\begin{aligned}&M_{Z^{'}}/g_{_B}\ge 6\,\mathrm{TeV}\;. \end{aligned}$$

(38)

In order to coincide with the experimental data, we choose $M_{Z^{'}}=4.2\,\mathrm{TeV}$ in our numerical analysis, then the scope of $g_{_B}$ is limited to $0<g_{_B}\le 0.7$. The LHC experimental data also constrain $\tan \beta ^{'}<1.5$. Considering the constraints from the experiments [80], for those parameters in Higgsino and gaugino sectors, we appropriately fix $M_1=500\, \mathrm{GeV},\;M_2=600 \,\mathrm{GeV},\;M_{BB^{'}}=500 \,\mathrm{GeV},\;M_{BL}=600\, \mathrm{GeV},\;\mu =700 \,\mathrm{GeV},\;\mu ^{'}=800 \,\mathrm{GeV}$. For simplify, we set $m_{H^\pm }=2\, \mathrm{TeV},\;B_\mu '=5\times 10^5\,\mathrm{GeV}^2,\;T_u=T_d=diag(1, 1, A_t)\, \mathrm TeV$. In addition, the first two generations of squarks are strongly constrained by direct searches at the LHC [81, 82] and the third generation squark masses are not constrained by the LHC as strong as the first two generations. Therefore we take $m_{\tilde{q}}^2=m_{\tilde{d}}^2=m_{\tilde{u}}^2=diag(2\mathrm{TeV},2\mathrm{TeV},m_{{\tilde{b}}})$, and the discussion about the observed Higgs singal in Refs. [34, 83] limits $m_{{\tilde{b}}}\gtrsim 1.5\,\mathrm{TeV}$.

It’s well known that the experimental observation on $Br(\bar{B}\rightarrow X_S \gamma )$ limits the relevant parameters strongly, hence we consider the constraint from ${\bar{B}}\rightarrow X_s\gamma$ in this work. In addition, we further consider the constraint from $B_s^0\rightarrow \mu ^+\mu ^-$, which might also limit our numerical analyses [84]. The latest experimental data for $Br({\bar{B}}\rightarrow X_S \gamma )$ and $Br(B_s^0\rightarrow \mu ^+\mu ^-)$ read [74]

$$\begin{aligned}&Br({\bar{B}}\rightarrow X_S \gamma )=(3.49\pm 0.19)\times 10^{-4},\end{aligned}$$

(39)

$$\begin{aligned}&Br(B_s^0\rightarrow \mu ^+\mu ^-)=(2.9_{-0.6}^{+0.7})\times 10^{-9}. \end{aligned}$$

(40)

Fig. 4

$Br(t\rightarrow c\gamma )$ (a), $Br(t\rightarrow cg)$ (b), $Br(t\rightarrow cZ)$ (c), $Br(t\rightarrow ch)$ (d) versus $g_{_B}$ for $g_{_{YB}}=-0.6\;(\mathrm{solid \;line}),\;g_{_{YB}}=-0.4\;(\mathrm{dashed \;line}),\;g_{_{YB}}=-0.2$ (dot-dashed line) are plotted. The dotted line denotes the MSSM predictions in the same parameter space

We also need to consider the constraint of SM-like Higgs boson mass [84]. Taking $\tan \beta '=1.1,\;g_{_B}=0.2,\;g_{_{YB}}=-0.6,\;A_t=-1.5$ and considering the restrictions from B physics and concrete Higgs boson mass, then letting $m_{{\tilde{b}}}$ runs from $1.5\,\mathrm{TeV}$ to $4\,\mathrm{TeV}$ and $\tan \beta$ runs from 2 to 40, the allowed region of them are

$$\begin{aligned}&10<\tan \beta<40,\;\;\;\;1700 \,\mathrm{GeV}<m_{{\tilde{b}}}<3500 \,\mathrm{GeV}. \end{aligned}$$

(41)

Then we plot $Br(t\rightarrow c\gamma ),\;Br(t\rightarrow cg),\;Br(t\rightarrow cZ)$ and $Br(t\rightarrow ch)$ versus $m_{{\tilde{b}}}$ in Fig. 3a–d, where the solid line, dashed line, dot-dashed line denote $\tan \beta =15,\;25,\;35$ respectively. From the picture, we can see that $Br(t\rightarrow c\gamma ),\;Br(t\rightarrow cZ)$ increase with the increasing of $m_{{\tilde{b}}}$ slowly, but $\tan \beta$ affects both of them obviously. And with the increasing of $\tan \beta$, they can reach $5\times 10^{-7}$, $4\times 10^{-7}$ respectively. However, Fig. 3b shows that $Br(t\rightarrow cg)$ has a sharp decrease when $\tan \beta =15$, and the turning point is around $1850 \,\mathrm{GeV}$. Due to the fact that the contributions from down type squarks to the branching ratio is cancelled by the contributions from charge Higgs boson at the turning point. In addition, from the mass matrix of down type squarks, we can see that the masses of down type squarks increase with the increasing of $\tan \beta$ or $m_{{\tilde{b}}}$, hence the turning point of $m_{\tilde{b}}$ decreases with the increasing of $\tan \beta$, which results in the turning point less than 1700 GeV when $\tan \beta =25,\;35$. The moving of turning point can be seen directly in Fig. 3d. The picture shows that the turning point decreases with the increasing of $\tan \beta$. In addition, with the increasing of $m_{{\tilde{b}}}$, the effect of $\tan \beta$ is more negligible to $Br(t\rightarrow ch)$. Since the main contribution to these processes come from down type squarks, $m_{{\tilde{b}}}$ affects the numerical results mainly through influencing the masses of the third generation down type squarks. Meanwhile, $\tan \beta$ not only presents in the diagonal sector of the mass matrix, but also dominates the off-diagonal sector, which indicates that $\tan \beta$ affects the numerical results mainly through influencing the mass of the down type squarks and the corresponding rotation matrix in the couplings involve down type squarks.

In order to see how new coupling constants $g_B$ and $g_{YB}$ in the B-LSSM affect $Br(t\rightarrow c\gamma )$, $Br(t\rightarrow cg)$, $Br(t\rightarrow cZ)$ and $Br(t\rightarrow ch)$, we continue to fix $\tan \beta =4,\;\tan \beta '=1.2,\;m_{{\tilde{b}}}=1.5\,\mathrm{TeV},\;A_t=-2$. Considering the limits from B physics and concrete Higgs mass, the allowed region of $g_{_B}$ and $g_{_{YB}}$ are

$$\begin{aligned}&-0.7<g_{_{YB}}<0,\;\;\;\;0.1<g_{_B}<0.7. \end{aligned}$$

(42)

Then we present $Br(t\rightarrow c\gamma ),\;Br(t\rightarrow cg),\;Br(t\rightarrow cZ)$ and $Br(t\rightarrow ch)$ varying with $g_{_B}$ in Fig. 4a–d respectively, where the three lines denote $g_{_{YB}}=-0.6$ (dotdashed line), $g_{_{YB}}=-0.4$ (dashed line) and $g_{_{YB}}=-0.2$ (dot-dashed line). In order to compare with the MSSM, we also plot the MSSM predictions in the same parameter space (dotted line). It can be noted that $Br(t\rightarrow cg)$ in the B-LSSM can exceed the MSSM prediction easily in our chosen parameter space. And $Br(t\rightarrow c\gamma ),\;Br(t\rightarrow cZ),\;Br(t\rightarrow ch)$ in the B-LSSM can exceed the MSSM predictions when $g_{_B}$ is small and $|g_{_{YB}}|$ is large. In addition, as $g_{_{YB}}$ approach to zero, all of the branching ratios depend on $g_{_B}$ negligibly, which indicates that the effect of $g_{_B}$ to these four processes is influenced by the strength of gauge kinetic mixing strongly. $g_{_B}$ and $g_{_{YB}}$ affect the numerical results mainly in three ways. Firstly, $g_{_B}$ and $g_{_{YB}}$ affect $Br(t\rightarrow c\gamma ,\;g,\;Z,\;h)$ by influencing the down type squark masses and the corresponding rotation matrix, which appears in the couplings involve the down type squarks. Secondly, they make new contributions to $Br(t\rightarrow cZ)$ by the $Z-Z'$ mixing. Thirdly, they affect the theoretical prediction on $Br(t\rightarrow ch)$ by mixing the Higgs doublets with the exotic singlets.

5 Summary

In the $U(1)_{B\text {--}L}$ extension of MSSM, under a minimal flavor violating assumption for the soft breaking terms, we focused on the top quark rare decay processes $t\rightarrow c\gamma , cg, cZ, ch$. Compared with the MSSM, new definition of the down type squark masses can affect the theoretical evaluation on these processes. In addition, the mixing in the scalar sector and Z-$Z'$ sector can also make new contributions to $t\rightarrow ch$ and $t\rightarrow cZ$ decay channel respectively. In our used parameter space, the numerical results show that all of these processes are well below the experiment limits. And $\tan \beta$ is a major parameter to the processes $t\rightarrow c\gamma , cg, cZ, ch$, the corresponding branching ratios can be $5\times 10^{-7}$, $2\times 10^{-6}$, $4\times 10^{-7}$, $3\times 10^{-9}$ respectively. Simultaneously, new gauge coupling constants $g_{_B},\;g_{_{YB}}$ in the B-LSSM can also affect the numerical results of $Br(t\rightarrow c\gamma ,\;cg,\;cZ,\;ch)$.