New signals for vector-like down-type quark in U(1) of $E_6$

Abstract

We consider the pair production of vector-like down-type quarks in an $E_6$ motivated model, where each of the produced down-type vector-like quark decays into an ordinary Standard Model light quark and a singlet scalar. Both the vector-like quark and the singlet scalar appear naturally in the $E_6$ model with masses at the TeV scale with a favorable choice of symmetry breaking pattern. We focus on the non-standard decay of the vector-like quark and the new scalar which decays to two photons or two gluons. We analyze the signal for the vector-like quark production in the $2\gamma +\ge 2j$ channel and show how the scalar and vector-like quark masses can be determined at the Large Hadron Collider.

1 Introduction

Elementary particle physics has been at crossroads of expecting a breakthrough to understanding what lies beyond the Standard Model (SM) for quite some time now. The Large Hadron Collider (LHC) discovery of the Higgs boson [1, 2] came much to the satisfaction of confirming the SM picture of the electroweak interactions. Although the reported hints of several phenomena that would have definitely indicated physics beyond the SM (BSM) have not survived further scrutiny by the LHC experiment, the diphoton excess at LHC [3,4,5] brought back the attention to heavy vector-like quarks and extended scalar sectors amongst many other models. The SM is widely believed to be an incomplete theory due to the lack of explanation to several outstanding issues (e.g. neutrino masses, dark matter candidate, etc.). The Grand Unified Theories (GUTs) are known to present novel ideas in addressing the above issues in the SM while also proposing to unify the three SM gauge couplings to one at a high scale. Most of the GUT models have testable consequences at the TeV scale which are in the form of an extra gauge group such as an extra U(1) and some additional new particles with heavy masses. We look at such an example in the $E_6$ GUT model where one gets down-type vector-like (VL) fermions charged under an extra U(1) gauge symmetry. In this work we focus on an interesting signal of the down-type vector-like quark (VLQ) at LHC.

Note that vector-like fermions exist in many BSM scenarios and a lot of phenomenological studies on the down-type VLQs exist in the literature [6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27]. The current experimental bounds on the mass of down-type VLQ are obtained under a certain assumption of its decay modes [28,29,30,31,32,33,34]. For a down-type VLQ the searches are based on the assumption that it decays to one of the SM final states Zb, Wt and bh. The current experimental lower bound on the mass of the down-type vector-like quark which mixes only with the third generation quark is around 730 GeV from Run 2 of the LHC [28] and is around 900 GeV from Run 1 of the LHC [31]. Similarly, the current lower bound for a vector-like quark which mixes with the light quarks is around 760 GeV from Run 1 of the LHC [34]. While strong limits can be derived from these conventional search channels, the bounds get relaxed once new non-standard decay modes are present and start dominating over the SM channels. In this work we discuss a non-standard decay channel of the down-type VLQ and about its possible signatures in a non supersymmetric version of $E_6$ model. A recent work discussing detailed phenomenology of vector-like quarks in $E_6$ model can be found in Ref. [35]. In our case, we look at the VLQs and singlet scalars which are particles already present in the $E_6$ GUT, as discussed later. Using appropriate symmetry breaking pattern, one U(1) in addition to the SM gauge symmetry remain unbroken at the TeV scale or even higher. The heavy down-type quark xd, which is a color triplet and an SU(2) singlet with an electric charge of $-1/3$, is pair produced dominantly from two gluons via strong interactions at the LHC. Also, three such xd and $\overline{xd}$ quarks naturally appear in our model based on $E_6$ from three fermion families. A singlet scalar is also naturally present which is responsible in breaking the additional U(1) at the TeV scale. The pattern of symmetry breaking that we shall use gives the singlet scalar mass which is close to the xd-quark mass. Our $E_6$ model will be discussed in the next section. The quantum numbers of all the particles are fixed from the $E_6$ symmetry. The VLQ has a dominant decay mode in the non-standard form of a SM quark and the new singlet scalar which is the focus of this study. We discuss the phenomenology of such a scenario and on the observable signatures for the vector-like down-type quarks at the LHC when the singlet scalar decays to a pair of photons or a pair of gluons. We shall have events with dijet/diphoton resonances at the same mass and these predictions can be tested as more data accumulates at the upcoming 13 TeV LHC run.

This paper is organized as follows. In Sect. 2 below, we discuss our model and the formalism. In Sect. 3, we discuss the phenomenology of our model. This gives emphasis on the prediction regarding the vector-like quarks through a new channel. The Sect. 4 contains our conclusions and discussions.

2 The model and formalism

We work with an effective symmetry at the TeV scale where the SM is augmented with an extra $U(1)'$. This extra $U(1)'$ is a special subgroup of the $E_6$ GUT [36,37,38,39,40,41,42,43,44,45,46]. We consider the non-supersymmetric version of $E_6$. The symmetry group $E_6$ is special in the sense that it is anomaly free, and having chiral fermions. Its fundamental representation decomposes under SO(10) as

$$\begin{aligned} \mathbf{27} = 16 +10 +1~. \end{aligned}$$

The representation 16 contains the 15 SM fermions, as well as a right-handed neutrino. It decomposes under SU(5) as

$$\begin{aligned} \mathbf{16} = 10 + \bar{5} +1. \end{aligned}$$

And the 10 representation decomposes under SU(5) as

$$\begin{aligned} \mathbf{10} = 5 +\bar{5}. \end{aligned}$$

The 5 contains a color triplet and an $SU(2)_L$ doublet, whereas $\bar{\mathbf{5}}$ contains a color anti-triplet and another SU(2) doublet, while the $\mathbf{1}$ is a SM singlet. The gauge bosons are contained in the adjoint $\mathbf{78}$ representation of $E_6$.

The full particle content of ${\mathbf{27}}$ representation, which contains the SM fermions as well as extra fermions, are shown in the first two columns of Table 1. For three families of the SM fermions, we use three such ${\mathbf{27}}$. The $E_6$ gauge symmetry can be broken as follows [47, 48]:

$$\begin{aligned} E_6 \rightarrow \ SO(10) \times U(1)_{\psi } \rightarrow \ SU(5) \times U(1)_{\chi } \times U(1)_{\psi }.\nonumber \\ \end{aligned}$$

(1)

The $U(1)_{\psi }$ and $U(1)_{\chi }$ charges for the $E_6$ fundamental $\mathbf{27}$ representation are also given in Table 1. The $U(1)'$ is a linear combination of the $U(1)_{\chi }$ and $U(1)_{\psi }$

$$\begin{aligned} Q^{\prime }= & {} \cos \theta \ Q_{\chi } + \sin \theta \ Q_{\psi }~.~\, \end{aligned}$$

(2)

The other orthogonal linear combination of $U(1)_{\chi }$ and $U(1)_{\psi }$ as well as the SU(5) are broken at a high scale. This will allow us to have a large doublet–triplet splitting scale, which prevents rapid proton decay if the $E_6$ Yukawa relations were enforced. This will require either two pairs of ($\mathbf{27}$, ${{\overline{\mathbf{27}}}}$) and one pair of ($\mathbf{351'}$, ${\overline{\mathbf{351}'}}$) dimensional Higgs representations, or one pair of ($\mathbf{27}$, ${{\overline{\mathbf{27}}}}$), $\mathbf{78}$, and one pair of ($\mathbf{351'}$, ${\overline{\mathbf{351}'}}$) dimensional Higgs representations (detailed studies of $E_6$ theories with broken Yukawa relations can be found in [49,50,51]). For our model, the unbroken symmetry at the TeV scale is $SU(3)_C \times SU(2)_L \times U(1)_Y \times U(1)'$.

Table 1 Decomposition of the $E_6$ fundamental $\mathbf{27}$ representation under SO(10), SU(5), and the $U(1)_{\chi }$, $U(1)_{\psi }$ and $U(1)'$ charges

We explain our convention in some details as given in Table 1. Our notation is similar to what is used in the supersymmetric case. We have denoted the SM quark doublets, right-handed up-type quarks, right-handed down-type quarks, lepton doublets, right-handed charged leptons, and right-handed neutrinos as $Q_i$, $U_i^c$, $D_i^c$, $L_i$, $E_i^c$, and $N_i^c$, respectively. In our model, we introduce three fermionic $\mathbf{27}$s, one scalar Higgs doublet field $H_u$ from the doublet of $\mathbf{{5}}$ of SU(5), one scalar Higgs doublet field $H_d$ from the doublet of $\bar{\mathbf{5}}$ of SU(5), one scalar SM singlet Higgs field T from the singlet of $\mathbf{16}$ of SO(10), and one scalar SM singlet Higgs field S from the singlet of $\mathbf{27}$ of $E_6$. Thus, similar to the fermions, all the scalars with masses at the TeV scale are coming from the $\mathbf{27}$ of $E_6$. Note that the new additional fermions from the $\mathbf{27}$ with masses at the TeV scale are $N_i^c$, $XD_i$, $XL_i^c$, $XD_i^c$, $XL_i$, and $XN_i$. For details see Table 2.

In our model, S gives the Majorana masses to the right-handed neutrinos $N^c_i$ after $U(1)'$ gauge symmetry breaking, i.e., the terms $S N_i^c N_i^c$ are $U(1)'$ gauge invariant. Thus, we obtain the low energy Type I seesaw mechanism at the renormalizable level. As is well known, in generic $U(1)'$ models from $E_6$, the terms $S N^c_i N^c_i$ are not $U(1)'$ gauge invariant. Therefore, the mixing angle in our model is given by

$$\begin{aligned} \tan \theta = \sqrt{5/3}. \end{aligned}$$

(3)

The complete scalar potential, including the fields $H_u, \, H_d, \, S$ and T, is

$$\begin{aligned} V=&-\mu _1^2 \,H_u^\dag H_u - \mu _2^2 \, H_d^\dag H_d + \lambda _1\, \left( H_u^\dag H_u\right) ^2 \nonumber \\&+ \lambda _2\, \left( H_d^\dag H_d\right) ^2 +\lambda _3\, \left( H_u^\dag H_u\right) \left( H_d^\dag H_d\right) \nonumber \\&+ \lambda _4\, [\epsilon _{ab}(H_u)_a (H_d)_b] \left[ \epsilon _{mn}\left( H_u^*\right) _m \left( H_d^*\right) _n\right] \nonumber \\&- m_S^2 |S|^2 - m_T^2 |T|^2 +\lambda _S |S|^4 +\lambda _T |T|^4 \nonumber \\&+ \lambda _{ST} |S|^2 |T|^2 + (\sigma S T^2 + H.C) \nonumber \\&+ \lambda _5 \, H_u^\dag H_u |S|^2 + \lambda _6 \, H_d^\dag H_d |S|^2 \nonumber \\&+ \lambda _7 \, H_u^\dag H_u |T|^2 +\lambda _8 \, H_d^\dag H_d |T|^2 \nonumber \\&+ (M\,[\epsilon _{ab}(H_u)_a (H_d)_b] S + H.C.) \nonumber \\&+ \left( \rho \,\left[ \epsilon _{ab}(H_u)_a (H_d)_b\right] T^* T^* + H.C.\right) . \end{aligned}$$

(4)

Table 2 The particles and their quantum numbers under the $SU(3)_C \times SU(2)_L \times U(1)_Y \times U(1)'$ gauge symmetry. Here, the correct $U(1)'$ charges are the $U(1)'$ charges in the above Table divided by $4{\sqrt{15}}$

Among the parameters in the potential, $\sigma$, M and $\rho$ are in general complex and all others are real. When considering the singlet part of the potential independently we note that without the term $\sigma S T^2$, there are two global U(1) symmetries for the complex phases of S and T. After S and T obtain the Vacuum Expectation Values (VEVs), we have two Goldstone bosons, and one of them is eaten by the extra U(1) gauge boson. Thus, to avoid the extra Goldstone boson, one needs the term $\sigma S T^2$ to break one global symmetry. This leaves us with only one U(1) symmetry in the above potential, which is the extra $U(1)'$ gauge symmetry. Thus, after S and T acquire the VEVs, the $U(1)'$ gauge symmetry is broken, and S and T will be mixed via the $\lambda _{ST} |S|^2|T|^2$ and $\sigma S T^2$ terms.

The SM gauge boson masses are determined by the VEVs of the SU(2) doublet scalars and therefore $v_{EW}=\sqrt{v_d^2 + v_u^2} \simeq 246$ GeV. The scalar fields with the VEVs are given by

$$\begin{aligned}&H_u = \begin{pmatrix} \phi _1^+ \\ \phi _1^0 \end{pmatrix},\quad H_d = \begin{pmatrix} \phi _2^0 \\ \phi _2^- \end{pmatrix}, \\&S = \frac{1}{\sqrt{2}}(v_s + s_1 + i s_2), \quad T = \frac{1}{\sqrt{2}}(v_t + t_1 + i t_2),\\&\phi _1^0 = \frac{1}{\sqrt{2}}\left( v_u + \phi _1^{0r} + i \phi _1^{0i}\right) ,\quad \phi _2^0 = \frac{1}{\sqrt{2}}\left( v_d + \phi _2^{0r} + i \phi _2^{0i}\right) . \end{aligned}$$

The structure for the VEVs is given as

$$\begin{aligned} \langle H_d \rangle&= \begin{pmatrix} v_d/\sqrt{2} \\ 0 \end{pmatrix},~&\langle H_u \rangle&= \begin{pmatrix} 0 \\ v_u/\sqrt{2}\end{pmatrix},~ \nonumber \\ \langle T\rangle&= v_t/\sqrt{2},~&\langle S \rangle&= v_s/\sqrt{2}. \end{aligned}$$

(5)

The tree level tadpole equation solutions help in expressing the parameters $\mu _1^2$, $\mu _2^2$, $m_S^2$ and $m_T^2$ in the potential in terms of the other parameters and are of the following form:

$$\begin{aligned} \mu _1^2&= \frac{1}{2v_u}\left( -\sqrt{2} M v_d v_s - v_d v_t^2 \rho + 2 v_u^3 \lambda _1 + v_d^2 v_u \lambda _3 \right. \nonumber \\&\quad \left. + v_d^2 v_u \lambda _4 + v_s^2 v_u \lambda _5 + v_t^2 v_u \lambda _7\right) ,\end{aligned}$$

(6)

$$\begin{aligned} \mu _2^2&= \frac{1}{2v_d}\left( -\sqrt{2} M v_u v_s - v_u v_t^2 \rho + 2 v_d^3 \lambda _2 + v_d v_u^2 \lambda _3 \right. \nonumber \\&\quad \left. + v_d v_u^2 \lambda _4 + v_s^2 v_d \lambda _6 + v_t^2 v_d \lambda _8\right) ,\end{aligned}$$

(7)

$$\begin{aligned} m_S^2&= \frac{1}{2 v_s} \left( - \sqrt{2} M v_d v_u + 2 v_s^3 \lambda _S + v_s v_t^2 \lambda _{ST} \right. \nonumber \\&\quad \left. + \sqrt{2} v_t ^2 \sigma + v_s v_u^2 \lambda _5 + v_d^2 v_s \lambda _6\right) ,\end{aligned}$$

(8)

$$\begin{aligned} m_T^2&= \frac{1}{2}\left( -2 v_d v_u \rho + v_s^2 \lambda _{ST} + 2 v_t^2 \lambda _T \sigma \right. \nonumber \\&\quad \left. + 2\sqrt{2} v_s + v_u^2 \lambda _7 + v_d^2 \lambda _8\right) \end{aligned}$$

(9)

The enlarged CP-even scalar mass squared matrix in the $(\phi _1^{0r}, \phi _2^{0r}, s_1, t_1)$ basis is given in Eq. (10) while the CP-odd scalar mass squared matrix in the basis $(\phi _1^{0i}, \phi _2^{0i}, s_2, t_2)$ is given in Eq. (11). We have

$$\begin{aligned} \mathscr {M}_{\text {even}}= & {} \left( \begin{array}{cccc} \frac{4 \lambda _1 v_u^3+\rho v_d v_t^2+\sqrt{2} \text {M} v_d v_s}{2 v_u} &{} -\frac{1}{2} \rho v_t^2-\frac{\text {M} v_s}{\sqrt{2}}+v_d v_u \left( \lambda _3+\lambda _4\right) &{} v_s v_u \lambda _5-\frac{\text {M} v_d}{\sqrt{2}} &{} v_t v_u \lambda _7-\rho v_d v_t \\ -\frac{1}{2} \rho v_t^2-\frac{\text {M} v_s}{\sqrt{2}}+v_d v_u \left( \lambda _3+\lambda _4\right) &{} \frac{4 \lambda _2 v_d^3+ \rho v_t^2 v_u+\sqrt{2} \text {M} v_s v_u}{2 v_d} &{} v_d v_s \lambda _6-\frac{\text {M} v_u}{\sqrt{2}} &{} v_d v_t \lambda _8-\rho v_t v_u \\ v_s v_u \lambda _5-\frac{\text {M} v_d}{\sqrt{2}} &{} v_d v_s \lambda _6-\frac{\text {M} v_u}{\sqrt{2}} &{} \frac{4 \lambda _s v_s^3-\sqrt{2} \sigma v_t^2+\sqrt{2} \text {M} v_d v_u}{2 v_s} &{} v_t \left( \sqrt{2} \sigma +v_s \lambda _{\text {ST}}\right) \\ v_t v_u \lambda _7- \rho v_d v_t &{} v_d v_t \lambda _8-\rho v_t v_u &{} v_t \left( \sqrt{2} \sigma +v_s \lambda _{\text {ST}}\right) &{} 2 v_t^2 \lambda _t \\ \end{array} \right) ,\end{aligned}$$

(10)

$$\begin{aligned} \mathscr {M}_{\text {odd}}= & {} \left( \begin{array}{cccc} \frac{v_d \left( \rho v_t^2+\sqrt{2} \text {M} v_s\right) }{2 v_u} &{} \frac{1}{2} \left( \rho v_t^2+\sqrt{2} \text {M} v_s\right) &{} \frac{\text {M} v_d}{\sqrt{2}} &{} -\rho v_d v_t \\ \frac{1}{2} \left( \rho v_t^2+\sqrt{2} \text {M} v_s\right) &{} \frac{\left( \rho v_t^2+\sqrt{2} \text {M} v_s\right) v_u}{2 v_d} &{} \frac{\text {M} v_u}{\sqrt{2}} &{} -\rho v_t v_u \\ \frac{\text {M} v_d}{\sqrt{2}} &{} \frac{\text {M} v_u}{\sqrt{2}} &{} \frac{\text {M} v_d v_u-\sigma v_t^2}{\sqrt{2} v_s} &{} -\sqrt{2} \sigma v_t \\ -\rho v_d v_t &{} -\rho v_t v_u &{} -\sqrt{2} \sigma v_t &{} 2 \rho v_d v_u-2 \sqrt{2} \sigma v_s \\ \end{array} \right) . \end{aligned}$$

(11)

It is quite clear that one can avoid any mixing between the doublet and singlet scalar sectors $(\phi _1^{0r}, \phi _2^{0r})$ and $(s_1,t_1)$ if all of the parameters $(\lambda _5, \, \lambda _6, \, \lambda _7, \, \lambda _8, \, M, \, \rho )$ are simultaneously taken to be zero. However, for such a choice the physical pseudoscalar from the $(\phi _1^{0i}, \phi _2^{0i})$ sector becomes massless. We however also note that for non-zero values of M and $\rho$ along with the following set of conditions:

$$\begin{aligned} \left\{ \lambda _5 = \frac{M v_d}{\sqrt{2}v_s v_u},\, \lambda _6 = \frac{M v_u}{\sqrt{2}v_s v_d},\, \lambda _7 = \frac{\rho v_d}{v_u},\, \lambda _8 = \frac{\rho v_u}{v_d}\right\} , \end{aligned}$$

(12)

we are able to generate mass for all the physical pseudoscalars but they also lead to zero mixing in the $(s_1,t_1)$ and $(\phi _1^{0r}, \phi _2^{0r})$ sectors. For simplicity throughout the paper the above set of conditions have always been ensured to avoid the mixing between the $(s_1,t_1)$ and $(\phi _1^{0r}, \, \phi _2^{0r})$ sectors. We will also consider the case where $M = 0$ and $\rho$ has nonzero value for illustration purposes. The mass eigenstates for the $(s_1, t_1)$ sector are represented by $s_h$ and $t_h$ with $s_h$ being the lighter one. Similarly for the $(\phi _1^{0r}, \, \phi _2^{0r})$ sector the mass eigenstates are $h_1$ and $h_2$, where $h_1$ always represents the experimentally observed Higgs boson with mass $\sim$ 125 GeV.

The charged scalar from the $(H_u,\,H_d)$ sector is represented by $h^-$ and its mass square value is given by

$$\begin{aligned} m_{h^-}^2 = \frac{\left( v_u^2+v_d^2\right) \left( \sqrt{2} M v_s + \rho v_t^2 -\lambda _4 v_d v_u\right) }{2 v_d v_u}. \end{aligned}$$

(13)

The fields $a_1$ and $a_2$ represent the physical pseudoscalars. In the limit $\{v_u, v_d, \, |\sigma |, \, v_s\} \ll v_t$ and $M=0$ the eigenvalues are given by

$$\begin{aligned} m_{a_1}^2 = \frac{\rho v_t^2 \left( v_u^2 + v_d^2\right) }{2 v_u v_d}\,\,\,\, \text {and} \,\,\,\,\,\,\,\,\, m_{a_2}^2 = -\frac{\sigma v_t^2}{\sqrt{2} v_s}. \end{aligned}$$

(14)

The relation between the gauge basis and the mass basis for the $(\phi _1^{0r},\phi _2^{0r})$ sector is given by

$$\begin{aligned} \begin{pmatrix} \phi _1^{0r} \\ \phi _2^{0r} \end{pmatrix} = \begin{pmatrix} \cos \theta _h &{} \sin \theta _h \\ -\sin \theta _h &{} \cos \theta _h \end{pmatrix} \begin{pmatrix} h_1 \\ h_2 \end{pmatrix}, \end{aligned}$$

(15)

where the mixing angle is

$$\begin{aligned} \sin 2\theta _h=\frac{2m_{12}}{\sqrt{(m_{11}-m_{22})^2+4m_{12}^2}} \end{aligned}$$

(16)

$$\begin{aligned} \cos 2\theta _h=\frac{-(m_{11}-m_{22})}{\sqrt{(m_{11}-m_{22})^2+4m_{12}^2}}, \end{aligned}$$

(17)

and the relation between the gauge basis and the mass basis for the $(s_1,t_1)$ sector is given by

$$\begin{aligned} \begin{pmatrix} s_1 \\ t_1 \end{pmatrix} = \begin{pmatrix} \cos \alpha &{} \sin \alpha \\ -\sin \alpha &{} \cos \alpha \end{pmatrix} \begin{pmatrix} s_h \\ t_h \end{pmatrix}, \end{aligned}$$

(18)

where the mixing angle is

$$\begin{aligned} \sin 2\alpha =\frac{2m_{34}}{\sqrt{(m_{33}-m_{44})^2+4m_{34}^2}} \end{aligned}$$

(19)

and

$$\begin{aligned} \cos 2\alpha =\frac{-(m_{33}-m_{44})}{\sqrt{(m_{33}-m_{44})^2+4m_{34}^2}} \,\, . \end{aligned}$$

(20)

The $m_{ij}$ are different components of the matrix $\mathscr {M}_{even}$. For illustration, with the given parameter values $(\lambda _1 = 0.2,\, \lambda _2 = 0.04,\, \lambda _3 = \lambda _4 = 0,\, M=0,\, \rho = 5\times 10^{-3},\, \frac{v_u}{v_d} = 2,\,v_s=950\,$ GeV, $\lambda _S=0.1, \, \lambda _T=7\times 10^{-3}, \, v_t=2\times 10^4 \,\text {GeV}, \, \lambda _{ST}=0.3,\,$ $\sigma _1=-17.9$ GeV), we get the following mass values of the scalars: $m_{h_1} = 125$ GeV, $m_{h_2} = 1582$ GeV, $m_{h^-} = 1581$ GeV, $m_{s_h}=600$ GeV, $m_{t_h}=3278$ GeV, $m_{a_1}=1581$ GeV and $m_{a_2}=2318$ GeV. The value of $\sin \alpha$ for the above parameter set is 0.7.

The Yukawa couplings in our model are

$$\begin{aligned} -\mathcal{L} = \,&y_{ij}^U Q_i U_j^c H_u + y_{ij}^D Q_i D_j^c H_d + y_{ij}^E L_i E_j^c H_d \nonumber \\&+ y_{ij}^N L_i N_j^c H_u \nonumber \\&+ y_{ij}^{XNd} XL_i^c XN_j H_d + y_{ij}^{XNu} XL_i XN_j H_u \nonumber \\&+ y_{ij}^{TD} D_i^c XD_j {T} \nonumber \\&+ y_{ij}^{TL} XL_i^c L_j {T} + y_{ij}^{SD} XD_i^c XD_j S \nonumber \\&+ y_{ij}^{SL} XL_i^c XL_j S \nonumber \\&+ y_{ij}^{SN} S N_i^c N_j^c + y_{ij}^{TXNN} T \, XN_i N_j^c \nonumber \\&+ \mathrm{H. C.}, \end{aligned}$$

(21)

where $i=1,~2,~3$. Thus, after S and T obtain VEVs or after $U(1)'$ gauge symmetry breaking, $(XD_i^c,~XD_i)$ and $(XL_i^c,~XL_i)$ will become vector-like particles from the $y_{ij}^{SD} XD_i^c XD_j S$ and $y_{ij}^{SL} XL_i^c XL_j S$ terms, and $(D_i^c,~XD_i)$ and $(XL_i^c,~L_i)$ will obtain vector-like masses from the $y_{ij}^{TD} D_i^c XD_j {T}$ and $y_{ij}^{TL} XL_i^c L_j {T}$ terms. After we diagonalize their mass matrices, we obtain the mixings between $XD_i^c$ and $D_i^c$, and the mixings between $XL_i$ and $L_i$.

We note that the $U(1)'$ gauge boson couples to all the SM fields in addition to the new matter and scalar fields. The covariant derivatives for the $SU(2)_L$ doublet and the singlet scalars are respectively given by

$$\begin{aligned} \begin{aligned} {\mathscr {D}}_{\mu }&= \left( \partial _\mu -i g \frac{\mathbf {\sigma }}{2}.\mathbf {W_\mu } - i g^\prime Y B_\mu - i g_{X} Y_{X} Z^{\prime }{_\mu }\right) , \end{aligned} \end{aligned}$$

(22)

where $Y(H_u)=\frac{1}{2}$, $Y(H_d)=-\frac{1}{2}$, $Y_X(H_u)=-\frac{2}{4\sqrt{15}}$, and $Y_X(H_d)=-\frac{8}{4\sqrt{15}}$;

$$\begin{aligned} \begin{aligned} {\mathscr {D}}_{\mu }&= (\partial _\mu - i g_{X} Y_{X} Z^{\prime }{_\mu }), \end{aligned} \end{aligned}$$

(23)

where $Y_X(S)=\frac{10}{4\sqrt{15}}$ and $Y_X(T)=-\frac{5}{4\sqrt{15}}$.

The mass square matrix for the neutral gauge boson sector in the $(W_3,B,Z^\prime )$ basis is then given as

$$\begin{aligned} \mathscr {M}= \left( \begin{array}{ll} &{} \mathscr {M}_{13} \\ (\mathscr {M}_{SM})_{2\times 2} &{} \mathscr {M}_{23} \\ \mathscr {M}_{13} \,\, \mathscr {M}_{23} &{} \mathscr {M} _{33} \\ \end{array}\right) , \end{aligned}$$

(24)

where

$$\begin{aligned} \mathscr {M}_{13}&=\frac{gg_X}{8\sqrt{15}}\left( 2v_u^2-8v_d^2\right) , \nonumber \\ \mathscr {M}_{23}&=-\frac{g^\prime g_X}{8\sqrt{15}}\left( 2v_u^2-8v_d^2\right) ~, \nonumber \\&\mathrm{and} \,\,\,\,\mathscr {M}_{33} = \frac{g_X^2}{240}\left( 4v_u^2+64v_d^2+25v_t^2+100v_s^2\right) . \end{aligned}$$

(25)

We can clearly see that the new gauge boson mass is dependent on the VEVs of all the scalars, such that one can choose one singlet VEV to be much smaller than the other and still have a very heavy $Z'$ that evades the existing limits. Moreover, the mixings between $W_3/B$ and $Z^\prime$ will be zero at tree level if $v_u=2v_d$.

The $(Z,Z')$ sector rotation matrix defined by

$$\begin{aligned} \begin{pmatrix} Z_1 \\ Z_2 \end{pmatrix}_{\text {mass basis}} = \begin{pmatrix} \cos \theta &{} \sin \theta \\ -\sin \theta &{} \cos \theta \end{pmatrix} \begin{pmatrix} Z \\ Z^{\prime } \end{pmatrix}, \end{aligned}$$

(26)

where the Z–$Z^\prime$ mixing angle is given by [40]

$$\begin{aligned} \theta = \frac{1}{2}\arctan \frac{2\Delta ^2}{ {M_{Z'}^2} - {M_{Z_0}^2} }, \end{aligned}$$

(27)

with

$$\begin{aligned} \Delta ^2&= \frac{g_X}{4\sqrt{15}} \sqrt{g^2+{g'}^2}\left( v_u^2-4v_d^2\right) , \\ {M_{Z_0}^2}&= \frac{1}{4}(g^2+{g'}^2)\left( v_u^2+v_d^2\right) , \\ {M_{Z'}^2}&= \frac{g_X^2}{240}\left( 4v_u^2+64v_d^2+25v_t^2+100v_s^2\right) . \end{aligned}$$

For the values of the parameters $(g_X=0.5, v_s = 1$ TeV, $v_t = 20$ TeV) and for $\frac{v_u}{v_d}$ lying in the range $0\le \frac{v_u}{v_d} \le 200$ we get $\theta \le 10^{-4}$ and a 3.2 TeV $Z^{\prime }$. This is expected in a model if there is atleast a $SU(2)_L$ singlet field which has a large VEV [40]. Such a small mixing is also necessary to avoid the constraints from electroweak precision data [52].

The mass matrix for the down-type quarks and the charged leptons in the $(q_1,q_2,q_3,xq_1,xq_2,xq_3)$ basis is given by

$$\begin{aligned} \frac{1}{\sqrt{2}} \begin{pmatrix} y_{ij}^D v_d &{} 0\\ y_{ji}^{TD}v_t &{} y_{ij}^{SD}v_s \end{pmatrix},&\frac{1}{\sqrt{2}} \begin{pmatrix} y_{ij}^E v_d &{} y_{ji}^{TL}v_t \\ 0 &{} y_{ij}^{SL} v_s \end{pmatrix}, \end{aligned}$$

(28)

where $i,j=1,2,3$. The $q_i$s and $xq_i$s represent the down-type quarks for the left matrix and charged leptons for the right matrix. These mass matrices would be diagonalized by a bi-unitary transformation which would lead to a mixing between the vector-like fermions and the SM fermions. However, one should note that the mixings between the left-handed fermions and the right-handed fermions will be dictated by different set of mixing angles. In our analysis we will allow mixings between the d quark and the first generation vector-like quark ($xd_1^0$) only and the mass matrix in the gauge basis $(d^0,xd_1^0)$ is given by

$$\begin{aligned} \frac{1}{\sqrt{2}} \begin{pmatrix} y_{11}^Dv_d &{} 0\\ y_{11}^{TD}v_t &{} y_{11}^{SD}v_s \end{pmatrix}\equiv \begin{pmatrix} m_1 &{} 0\\ m_2 &{} m_3 \end{pmatrix}. \end{aligned}$$

The mixing matrices which transform the gauge eigenstates $(d^0,xd_1^0)$ to mass eigenstates$(d,xd_1)$ are given by

$$\begin{aligned} S_i = \begin{pmatrix} \cos \theta _i &{} -\sin \theta _i \\ \sin \theta _i &{} \cos \theta _i \end{pmatrix},\quad \text {where}\,\,\, i = L,R, \end{aligned}$$

(29)

with the following left- and right-handed mixing angles:

$$\begin{aligned}&\sin 2\theta _L=\frac{2m_1m_2}{\sqrt{\left( m_1^2-m_2^2-m_3^2\right) ^2+4m_1^2m_2^2}}, \\&\cos 2\theta _L=\frac{-\left( m_1^2-m_2^2-m_3^2\right) }{\sqrt{\left( m_1^2-m_2^2-m_3^2\right) ^2+4m_1^2m_2^2}}, \\&\sin 2\theta _R=\frac{2m_2m_3}{\sqrt{\left( m_1^2+m_2^2-m_3^2\right) ^2+4m_2^2m_3^2}}, \\&\cos 2\theta _R=\frac{-\left( m_1^2+m_2^2-m_3^2\right) }{\sqrt{\left( m_1^2+m_2^2-m_3^2\right) ^2+4m_2^2m_3^2}}. \end{aligned}$$

We should also point out a few useful assumptions that we think are relevant for the analysis:

• 1. We have neglected any mixing between the electroweak doublet scalars and singlet scalars.

• 2. We also ensure that the new $U(1)'$ gauge boson does not have a significant mixing with the SM gauge boson Z ($\mathscr {M}_{13}, \mathscr {M}_{23}<< \mathscr {M}_{33}$).

• 3. For simplicity, we will take all types of Yukawa couplings $y^A_{ij}$ to be zero for $i\ne j$, for e.g. where $A\equiv TD, \, TL,$ $SD, \, SL$ (see Eq. (21)).

• 4. The mixing angles between the left-handed SM fermions and the vector-like fermions are taken to be very small, i.e., we assume the left-mixing angle $\theta _L\sim 0$ to avoid the flavor physics constraints [53]. To achieve this we assume $y_{ij}^{SD} \langle S \rangle>> y_{ij}^{TD} \langle {T}\rangle$ and $y_{ij}^{SL} \langle S \rangle>> y_{ij}^{TL} \langle {T}\rangle$. For the choice of the set of parameter values {$\frac{y_{11}^Dv_d}{\sqrt{2}}\sim m_d$, $\frac{y_{11}^{SD}v_s}{\sqrt{2}}\sim 640$ GeV, $v_t\sim 10^4$ GeV, $y_{11}^{TD}\sim 10^{-5}$} we get small values of mixing angles, i.e., $\sin \theta _L\sim 10^{-10}$ and $\sin \theta _R\sim 10^{-4}$.

• 5. A quick look at particle spectrum and the Yukawa terms in the Lagrangian involving the neutrinos suggest that there are five two component neutrino fields in the model (for each generation). Taking into account all possible mass terms for the neutrino sector we find that light neutrino mass ($\sim 0.01$ eV) can be generated, while the remaining Majorana neutrinos are all heavier than 500 GeV for $y_{11}^N = 10^{-3}$, $\frac{y_{11}^{SN}v_s}{\sqrt{2}} = 1$ TeV, $\frac{y_{11}^{SL}v_s}{\sqrt{2}} = 500$ GeV, $y_{11}^{XNd} = 10^{-2}$, $y_{11}^{XNu} = 10^{-2}$, $y_{11}^{TXNN} = 0.4$, $y_{11}^{TL} = 0$, $v_s = 950$ GeV, $v_t = 10^4$ GeV, $v_u = 2 v_d$.

3 Signals for vector-like quarks

The new VLQ will be dominantly produced via strong interaction, with subleading contributions coming from the s-channel exchange of the $\gamma , Z$ and $Z^\prime$. In situations where the VLQ mass is less than $M_{Z'}/2$, then the $Z^\prime$ mediated process can give a resonant contribution. However, these contributions are found to be not very significant. We list the various production mechanisms of the VLQ in Fig. 1. Note that one can in principle also produce the VLQs singly but they would be heavily suppressed as the production strength would depend on the mixing between the VLQs and SM quarks.

Fig. 1

The leading-order Feynman diagrams for the subprocess contributing to the pair production of the VLQ at the LHC

Fig. 2

The pair production cross section of $xd_1$ at the LHC with $\sqrt{s}=13$ TeV as a function of $M_{xd_1}$

Table 3 The couplings of VLQ $xd_1$ and d-quark with SM gauge bosons and, with the scalars $s_h$ and $t_h$. Coupling with gauge bosons are of the form $K\gamma ^{\mu }(c_V-c_A \gamma ^{5})$ and with scalars are of the form $K(c_S-c_P\gamma ^{5})$. Here $\alpha$ is the scalar sector mixing angle in Eq. (18), $\theta _L$ and $\theta _R$ are left and right mixing angles in Eq. (29)

In Fig. 2 we show the pair production cross section of the VLQ $xd_1$ as a function of its mass at both run-1 and current run of the LHC with $\sqrt{s}=13$ TeV. With a few 100 femtobarns of cross section, it would be highly unlikely for the LHC to miss the signal for VLQs when they decay directly to SM particles. These already put strong limits on the mass of the VLQs. However, a new decay mode for the VLQ can definitely alter the search strategies for these exotics even when the rates are significantly high.

With the details of the model discussed in the previous section, it is now possible to write down the interaction vertices for the VLQ and new scalars with the SM particles that we use in our calculation and analysis. We list the relevant interactions in Table 3.

The possible decay modes for a down-type VLQ in our model are to the SM particles given by $xd_1 \rightarrow h_1 \,d, \,d \,Z, \,$ and $\, u \,W^{-}$ while the non-standard decay modes would be $xd_1 \rightarrow s_h\,d,\,t_h \,d, \,a_1 \,d, \,a_2 \,d, \,h_2\,d, \,u \,h^- \, \text {and} \,\,d \,Z^\prime$. Here $h_1$ is the SM Higgs boson, $h_2$ is another CP-even scalar from the $(H_u,\,H_d)$ sector, u and d are SM first generation quarks, Z and $W^-$ are SM gauge bosons. $h^-$ is the charged scalar from the $(H_u,\,H_d)$ sector. $s_h$ and $t_h$ are CP-even scalars from (S, T) sector while $a_1$ and $a_2$ are CP-odd scalars. For simplicity we focus on the case where out of all the SM down-type quarks, $xd_1$ interacts only with the d quark through the Yukawa interaction with S and T. For a very small mixing between $xd_1$ and d quark and for a $Z'$ heavier than $xd_1$, the dominant decay modes of $xd_1$ become $s_h \, d, \,\, t_h\,d$, $a_1\,d$ and $a_2\,d$. The other decay modes are suppressed because the interaction strengths for these decays are proportional to the very small mixing angles $\sin \theta _L$ and $\sin \theta _R$ (Table 3). As discussed in the previous section and to be safe from flavor constraints, one can impose small mixing angles, for example, $\sin \theta _L\sim 10^{-10}$ and $\sin \theta _R\sim 10^{-4}$ as mentioned for a set of parameter choices of the model. This will ensure that the vector-like fermions do not decay to the SM gauge bosons and light SM fermions [54]. The decay to the SM Higgs and light down-type quark is again very suppressed, due to the coupling strength being proportional to $\sin \theta _R$ and mass of the down-type SM quark. The mixing in the Higgs sector has been neglected as a convenient choice to keep the number of free parameters to tune to be small.

Fig. 3

a Illustrating the diphoton branching ratio for $s_h$ decay as a function of the vector-like lepton mass for two different values of the lightest VLQ mass ($M_{xd_1}$). b The on-shell $s_h$ production cross section at LHC with $\sqrt{s}=13$ TeV through gluon fusion as a function the singlet VEV $v_s$ with $m_{s_h}=600$ GeV. For the above plots, we fix $M_{xd_2}\,=\,M_{xd_3}\,=$ 1.5 TeV, $y_{ii}^{SL}\,=\,1$. For a $M_{x\ell _1}\,=\,M_{x\ell _2}\,=\,M_{x\ell _3}\,=\,M_{x\ell }$, while for b $y_{11}^{SD}\,=\,1$, $M_{xd_1}=y_{11}^{SD}v_s/\sqrt{2}$, and $y_{22}^{SD}\,=\,\sqrt{2}\,M_{xd_2}/v_s$

The Z–$Z^\prime$ mixing which is anyhow strongly constrained by electroweak data in any U(1) extension beyond the SM is also negligible [52], thus avoiding the decay of VLQ to $d\,Z$ final state through this mixing. All other possible scalars other than $h_1$ from the doublet sector $(H_u, \, H_d)$ are heavier than the VLQ, and thus ensure absence of the decay of VLQ to them. So if $a_1$, $a_2$ and $t_h$ are also heavier than $xd_1$, the VLQ $xd_1$ decays to the lone $s_h \,d$ final state. This decay is not suppressed due to a direct Yukawa coupling of the SM quark and VLQ with T as well as the mixing between the $xd_1$ and d in the right-handed sector. Thus, the decay is made possible through not only the mixing between the CP-even components of the scalars S and T but also depends on $\sin \theta _R$. It turns out that even with a choice of the Yukawa strength of $10^{-5}$ or lower (where $\sin \theta _L\sim 10^{-10}$ and $\sin \theta _R\sim 10^{-4}$), this decay is still the dominant channel. Thus, with the minimal assumptions that mixing of the new states with the SM sector being small and negligible allows a very specific decay channel for the VLQ in the model.

Once the VLQ is produced at the LHC, it will almost always decay into the non-standard channel to give a light quark jet and the scalar $s_h$. The $s_h$ then decays promptly to either SM particles or any lighter states of the new particles in the spectrum. The decay modes for the scalar can be summarized as $s_h \rightarrow \overline{\ell _i} \,\ell _j, \overline{x\ell _i} \,x\ell _j, \,\overline{x\ell _i} \, \ell _j, \, \overline{\ell _i} \,x\ell _j, \, \gamma \, \gamma , \, g \,g, \overline{d} \, d, h_1 h_1$. Here $\ell _i$s are SM charged leptons and $x\ell _i$s are vector-like leptons. We avoid the decay of $s_h$ to a pair of VLLs by setting their mass such that $M_{x\ell _i}>m_{s_h}/2$. Here $M_{x\ell _i}$ is the mass for the ith generation vector-like lepton and $m_{s_h}$ is the mass of the scalar $s_h$. The Yukawa coupling $y_{ij}^{TL}$ has been chosen zero to avoid mixing between the VLL and the SM lepton sector, and thus avoiding the decay of $s_h$ to the final states $\overline{x\ell _i} \, \ell _j$, $\bar{\ell _i} \, x\ell _j$ and $\overline{\ell _i} \, \ell _j$. Additionally the decay to $\overline{d}d$ is controlled by the mixing angle $\sin \theta _L$ and is therefore too suppressed. With $xd_1$ heavier than $s_h$ the decay mode $s_h \rightarrow \overline{xd_1}\,d$ is not allowed. For the choice $M = 0$ the decay of $s_h$ to two SM Higgs boson final state $h_1 h_1$ depends on the trilinear coupling which is given by

$$\begin{aligned} g_{s_hh_1h_1}=\frac{(-\sin \alpha )\rho v_t \left( \cos \theta _h + \frac{v_u}{v_d}\sin \theta _h\right) ^2}{2\frac{v_u}{v_d}}, \end{aligned}$$

(30)

where the angles $\theta _h$ and $\alpha$ are defined in Eqs. (15) and (18), respectively. Using Eqs. (16) and (17) for the parameter values $(M=0,\,\lambda _3=0,\,\lambda _4=0,\,\sin \alpha =0.5,\,10^{-3}\le \rho \le 1,\,v_t=10^4\,\text {GeV}, 2\le \frac{v_u}{v_d}\le 100)$ we found that the absolute value of $g_{s_hh_1h_1}$ lies in the range $10^{-2}-10^{-10}$ ($\lambda _1$ and $\lambda _2$ has been adjusted accordingly to get $m_{h_1} \approx 125$ GeV). Based on this we can choose the parameters to avoid the decay of $s_h$ to $h_1h_1$ final state. Note that we have avoided mentioning the additional decay modes to scalar pairs besides the SM-like Higgs and the invisible decay of $s_h$, which are either kinematically forbidden or very suppressed with our choice of parameters. Hence the only allowed final states for $s_h$ decay are $\gamma \gamma$ and gg. Of course, the possibility for the di-Higgs mode exists and is an interesting possibility which is planned for future work which would include a more detailed analysis of the scalar and neutrino sector of the model with all mixings allowed.

Decay to $\gamma \gamma$ and gg will occur through the effective one-loop induced coupling. All of the three generations of down-type VLQs and charged VLLs will affect the branching ratios of $s_h$ to $\gamma \gamma$ and to gg. The coupling of $s_h$ to gluons and photons follows the standard notations that are being used in the literature and for clarification we are giving only the coupling with gluons by the effective Lagrangian

$$\begin{aligned} \mathscr {L}_{s_hGG} = - \lambda _{sgg} s_h ~G_{\mu \nu } G^{\mu \nu }~, \end{aligned}$$

(31)

with the effective coupling $\lambda _{sgg} = \alpha _s F_{1/2} (\tau _{xd}) /( 16 \pi v_s)$ where

$$\begin{aligned} F_{1/2}(\tau _{xd}) = 2(\tau _{xd}+(\tau _{xd}-1)f(\tau _{xd}))\tau _{xd}^{-2} \end{aligned}$$

(32)

represents the loop function and $f(\tau _{xd})=(sin^{-1}\sqrt{\tau _{xd}})^2$ with $\tau _{xd}=m_{s_h}^2/4M_{xd}^2 \,\, < 1$. Here, we have shown the contribution to the coupling from only one vector-like quark.

We plot the branching ratio for the scalar $s_h$ decaying into a pair of photons in Fig. 3a as a function of the VLL masses for two values of the lightest VLQ mass($M_{xd_1}$), while the other two VLQs have masses at 1.5 TeV. As the masses of light vector-like leptons are not severely constrained by experiments, we shall consider the results with all the three VLLs contributing to the diphoton decay. Here the mass values of all the three vector-like leptons have been taken to be same ($M_{x\ell }$), while the Yukawa couplings of $s_h$ to VLLs have been taken to be unity ($y_{ii}^{SL}=1$).

Note that the branching of the $s_h \rightarrow \gamma \gamma$ is very similar to the order at which the SM Higgs decay happens but slightly higher. This is because of the contributions of the VLLs which do not contribute to the $s_h \rightarrow gg$ mode. However, the decay to $\gamma \gamma$ mode is still between 0.5%–0.6% at best while the remaining decay probability is made up by the gg channel.

Figure 3b shows the production cross section of a 600 GeV $s_h$ at $\sqrt{s}=13$ TeV center of mass energy as a function of the singlet VEV $v_s$. The $s_h$ is being produced by the loop induced effective coupling in Eq. (31). For the cross section estimates we have chosen the Yukawa coupling of $s_h$ to the 1st generation VLQ $xd_1$ to be 1. The mass of $xd_1$ depends on the value $v_s$ following the relation $M_{xd_1}=y_{11}^{SD}v_s/\sqrt{2}$. The masses for the $xd_2$ and $xd_3$ have been taken to 1.5 TeV. The Yukawa couplings ($y_{22}^{SD}$ and $y_{33}^{SD}$) have different values for different values of $v_s$. Note that the $s_h$ production crucially depends on the Yukawa coupling, which can be tuned to control its production rate. In fact, this possibility was earlier used by us [55, 56] and in Ref. [57] with top-like VLQ to arrive at the possible explanation of the now hitherto disproved diphoton excess for a 750 GeV resonance [3,4,5].

As pointed out earlier, the VEVs for S and T, which are given by $v_s$ and $v_t$, respectively, play a significant role in giving mass to $Z^\prime$. We choose the mass of $Z^\prime$ to be above 4.2 TeV to avoid the constraints from the dilepton searches at LHC [58]. As can be seen from the mass square matrix of the neutral gauge boson sector (Eq. (25)), by choosing large values of $v_s$ or $v_t$, it is possible to avoid mixing between the SM and new gauge boson sector. But the vector-like lepton masses are given by the relation, $M_{x\ell _i}=y_{ii}^{SL}v_s/\sqrt{2}$. And $y_{ii}^{SL}$ is the Yukawa coupling of $s_h$ to the VLLs, which enters in the $s_h\gamma \gamma$ effective coupling. For a sub-TeV vector-like lepton and with significant value of Yukawa coupling it will not be possible to choose a very large value of $v_s$. We therefore choose a higher value for $v_t$ at 30 TeV, which effectively suppresses any significant mixing in the neutral gauge boson sector.

Note that after the decay of the VLQs there will be two $s_h$ and two jets in the final state. As the $s_h$ decays to two gluons or to two photons only, with almost 99% to the gluonic jets, the resultant final states are either $2\gamma +4j$, $4\gamma +2j$ or 6j. The cross section for the $4\gamma \,+\,2j$ final state is quite small while the QCD background for the 6j final state is significantly large compared to the $2\gamma \,+\,4j$ final state. Thus, these two channels would require large statistics to leave any imprint of their signal at the LHC. So in all likelihood the remaining channel of $2\gamma +4j$ seems the most promising channel which we shall focus on for our analysis.

For the $2\gamma +4j$ final state one of the $xd_1$ will eventually have a full hadronic decay to three jets. The bound on the branching ratio for the decay of a color triplet vector-like quark to three jets for different masses of vector-like quark has been obtained in [59] using the existing searches for resonances in multijet final states by the CDF Collaboration [60] at the Tevatron, and by the CMS and ATLAS Collaborations at the LHC using data from the 7 TeV [61, 62], 8 TeV [63, 64] and 13 TeV [65] runs. Reference [59] shows that the physical region where BR(VLQ $\rightarrow 3j)\le 1$ for different mass values of vector-like quark remains unconstrained except for a tiny region around 500 GeV.

Table 4 Two benchmark scenarios. The cross section is evaluated at the 13 TeV LHC. Note that $y_{ii}^{SD}\,=\,\sqrt{2}\,M_{xd_i}/v_s$, $y_{ii}^{TD}\,=\, 10^{-6}$ and we fix $M_{xd_2}=M_{xd_3}=1.5$ TeV

To analyze the signal in our model, we note that the hardness of the jet from the decay $xd_1 \rightarrow s_h \, j$ will depend on the mass difference between the VLQ $xd_1$ and the scalar $s_h$. This will affect the signal efficiency in the $2\gamma +4j$ channel as well as dictate how well the mass reconstruction for the parent particles can be made. We will discuss these features by considering two benchmark scenarios with different mass gaps between the $xd_1$ and $s_h$, where in one case the jet is hard while for the other case the jet would be comparatively soft. We choose BP1 ($M_{xd_1}=640$ GeV, $m_{s_h}=600$ GeV) with a small mass difference between the VLQ and $s_h$ as well as BP2 ($M_{xd_1}=850$ GeV, $m_{s_h}=600$ GeV) with a large mass difference of 250 GeV between them. For the two benchmarks the masses of the VLQs from the other two generations have been kept at 1.5 TeV. By comparing the cross sections for an 850 GeV VLQ with that of a 1.5 TeV VLQ from Fig. 2 it can be concluded that the two VLQs having mass of 1.5 TeV will not contribute to the final state of the analysis. Other benchmark details including the pair production cross section for $xd_1$ and the branching fractions for the decay of $s_h$ are shown in Table 4. We also check that the current upper limits on the cross section for the diphoton production through a narrow-width scalar resonance at 13 TeV run of LHC given by CMS Collaboration [66] is satisfied for our choice of the benchmark points. The upper limits on the cross section for the dijet production through a narrow-width resonance at the 13 TeV LHC, given by the CMS Collaboration [67, 68] are also satisfied.

Note that for BP1 the mass difference between $xd_1$ and $s_h$ is 40 GeV. This would mean that the jet coming from the decay of $xd_1$ is quite soft. Although at a hadronic machine such as the LHC, the jet multiplicity from parton showering would be invariably increased, we intend to focus on relatively hard jets and therefore would like to neglect soft jets in the process. So for the analysis of BP1 we consider a final state with smaller jet multiplicity given by $2\gamma \,+ \, \ge \,2j$. We demand that the jets have at least a minimum 40 GeV transverse momentum. The dominant SM background for such a final state is through all subprocesses contributing to $pp\rightarrow 2\gamma \,+ \, \ge \,2j$ (with $p_T(j) \, > \, 40$ GeV). For BP2 where the mass gap between $xd_1$ and $s_h$ is above 200 GeV, one expects the jet from $xd_1$ decay to be quite hard and thus the SM background is given by $pp \rightarrow 2\gamma \,+ \, \ge \,4j$.

We have implemented the TeV-scale $U(1)^\prime$ extended model derived from $E_6$ GUT in LanHEP [69, 70] to generate the model files for CalcHEP [71]. Using the model files we generated events for the pair production of VLQs (in LHEF format [72]) at the LHC with $\sqrt{s}=13$ TeV and the subsequent decays of the $xd_1$ and $s_h$ were included as a decay table for the model (in SLHA format [73, 74]) with the help of CalcHEP. We then use these files to decay the unstable particles, and pass the generated parton-level events for showering and hadronization in PYTHIA 8.2 [75]. To enable detector simulation, we then linked the HepMC2 [76] libraries with PYTHIA 8.2 to translate PYTHIA 8 events into HepMC format. For simulating the background, we generated the events at leading-order accuracy using MadGraph5 [77]. Pythia 6 [78] interfaced in MadGraph5 was used for parton showering and hadronization of the background events, and to get event files in STDHEP format.

For both signal and background we include the detector effects and have reconstructed the final state objects using DELPHES 3 [79]. These are obtained in a CMS environment. Further, FastJet [80] embedded in DELPHES has been used to reconstruct the jets. In the DELPHES framework the anti-$k_T$ algorithm with a cone size 0.5, $p_T^{j}>20$ GeV and $|\eta _{parton}|<2.5$ is used to reconstruct the jets. The phenomenological event-analysis is done with the MadAnalysis5 package using the event format ROOT.

In case of BP1 for which the mass difference between the lightest vector-like quark ($xd_1$) and the scalar ($s_h$) is small, we have generated $pp\rightarrow 2\gamma +2j$ events as background at 13 TeV LHC. At the level of generation of events certain basic cuts have been imposed on the final state particles. All jets and photons satisfy $|\eta | < 2.5$ and each final state particle is separated from all other final state particles with an angular separation ($\Delta R$) value greater than 0.4. The transverse momenta of photons and jets satisfy

$$\begin{aligned} p_T(j) \,> \, 20 \,\, \text {GeV} \quad \text {and} \quad p_T(\gamma ) \,> \, 100 \,\, \text {GeV}. \end{aligned}$$

(33)

The final state photons for the signal come from the decay of the $s_h$, which has 600 GeV mass and hence the probability for the photons for the signal to have higher $p_T$ values is more compared to the background. Hence a 100 GeV $p_T$ cut for photon has been used for the generation of background because the phase space with lower photon $p_T$ will be largely populated by background compared to the signal. For 13 TeV LHC, with the above cuts taken into account and at leading-order (LO) accuracy the value of the cross section for the parton-level background for BP1 is around 234 fb.

For BP2 where the mass difference between $xd_1$ and $s_h$ is 250 GeV, $pp \rightarrow 2\gamma +4j$ events have been generated as the background. The basic cuts on the pseudo-rapidity ($\eta$) and on the angular separation ($\Delta R$) of the final state particles have been taken to be same as that of the benchmark BP1. The cut on the photon $p_T$ is taken to be the same 100 GeV as in the case of BP1. We have imposed different $p_T$ cuts on the four jets and those are given by

$$\begin{aligned} p_T(j_1) \,> \, 80 \,\, \text {GeV}, \,\,\,\,\,\,\,\,\,\,\,\,\,\,\,&p_T(j_2) \,> \, 80 \,\, \text {GeV}, \nonumber \\ \,\,\,\,p_T(j_3) \,> \, 40 \,\, \text {GeV} \,\,\,\,\,\,\,\text {and}&p_T(j_4) \, > \,40 \,\, \text {GeV}. \end{aligned}$$

(34)

With the above cuts the cross section at the parton level for the background, for BP2 i.e. for the process $pp \rightarrow 2\gamma +4j$, comes out to be around 12.15 fb. Similarly the signal events $pp \rightarrow xd_1 xd_1$ have been generated using the event generator CalcHEP. The pair production cross sections are shown in Table 4.

For the reconstructed events we choose the following selection criteria on the photons, jets and leptons:

• A jet is considered in an event if $p_T(j)>40$ GeV and $|\eta (j)|<2.5$.

• An electron or a muon is considered in the lepton set if $p_T(\ell )>10$ GeV and $|\eta (\ell )|<2.5$.

• A photon is considered in an event if $p_T(\gamma )>40$ GeV and $|\eta (\gamma )|<2.5$.

• All final state candidates are separated from each other with a minimum angular separation satisfying $\Delta R > 0.4$.

The preselection criteria for BP1 is to consider events having 2 photons and a minimum of two jets. For BP2 the preselection criteria is to consider events with two photons and a minimum of four jets in the final state. We vetoed all the events having at least one isolated lepton of $p_T$ value greater than 10 GeV. For the two leading (in $p_T$) jets and the two leading photons in the signal it is expected that they come from the decay of the VLQ. As the mass of the VLQ for the two cases is above 600 GeV, the two leading jets will have a large amount of $p_T$ compared to the leading jets in the background. So for both benchmarks we have considered the events with the leading two jets and two photons with $p_T$ value greater than 100 GeV.

To further increase the signal-to-background ratio we apply the following selection cuts to the analysis for BP1:

$$\begin{aligned}&p_T(\gamma _1)\ge 200 \, \text {GeV}, \quad p_T(j_1)\ge 150 \, \text {GeV}, \nonumber \\ {}&M_{eff}\ge 800 \, \text {GeV}. \end{aligned}$$

(35)

The cut-flow table for BP1 signal and background is shown in Table 5. Note that to generate the background events with large statistics we have used the preselection cuts given in Eq. (33).

Similarly for BP2 we have besides the criterion in Eq. (33), the additional preselection requirements for jets as given by Eq. (34) to generate the SM background with good statistics. We further impose the stronger selection cuts on the events

$$\begin{aligned} p_T(\gamma _1)\ge 200 \, \text {GeV}, \qquad \quad M_{eff}\ge 1000 \, \text {GeV}, \end{aligned}$$

(36)

which help in improving the signal-to-background ratio for the signal events in the $2\gamma + 4j$ final state.

With these cuts and a 100 fb${^{-1}}$ integrated luminosity we get a statistical significance of 5$\sigma$ for BP1 as can be seen from Table 5. For BP2 we get a significance of 1.9$\sigma$ as can be seen from Table 6. The significance is calculated using the formula $\sigma = \sqrt{\frac{S}{S+B}}$.

Table 5 The selected events after each step of selection criteria for BP1 with an integrated luminosity of 100 $\hbox {fb}^{-1}$

Table 6 The selected events after each step of selection criteria for BP2 with an integrated luminosity of $100 \,\hbox {fb}^{-1}$

Fig. 4

The normalized invariant mass distributions for the two leading photons of benchmark scenarios BP1 and BP2

As we trigger upon two hard photons in the final state which come from the decay of the scalar $s_h$, the mass of the scalar $s_h$ can be reconstructed by looking at the invariant mass distribution of the two leading photons for both BP1 and BP2. To highlight this we plot the normalized invariant mass distribution of the two leading photons for both signal and background in Fig. 4. As expected the signal from the pair production of the VLQ is confined to a bin around the mass of the scalar $s_h$ with a clear peak for the signal at $m_{s_h}=600$ GeV. There would in principle also be an invariant mass peak for a jet pair around the $s_h$ mass, which, however, is more challenging to observe due to the large spread in their invariant mass distribution.

Similarly, to reconstruct the mass of the VLQ one can use the fully hadronic channel giving three jets or the semi-hadronic channel giving two photons and a jet. With the knowledge of the reconstructed mass for $s_h$ through the $2\gamma$ invariant mass peak, the mass for the vector-like quark can be reconstructed for both BP1 and BP2. To compare the reconstruction of the VLQ in the two channels, we first plot the 3j invariant mass distribution comprised of the leading jets in the events for both the signal and the background in Fig. 5. Although a distinct excess in the distribution exists around the mass of VLQ for both BP1 and BP2, the spread is quite wide and hence unclear as a mass resonance.

Fig. 5

The invariant mass distribution for the leading three jets for both benchmark scenarios BP1 and BP2

Fig. 6

The average of normalized invariant mass distributions of the leading two photons with a jet. The average for BP1 starts from the third leading jet and the average for BP2 starts from the first leading jet

Although the other channel with two photons and a hard jet should be a much cleaner and more precise mode to reconstruct the parent VLQ mass, it does suffer from the ambiguity of pairing the right jet with the pair of photons. In addition, for BP1 the mass splitting between the $xd_1$ and $s_h$ is quite small and therefore the choice of the right jet is affected by other soft jets that may originate from showering and fragmentation effects. To account for this ambiguity, we use the primary information on the kinematic characteristics of events for both BP1 and BP2 that is available to us to determine how we should combine the jets with the two photons. Owing to the small mass gap in BP1, we can safely assume that the two leading jets for the signal in BP1 would come from the decay of $s_h$ and therefore can be safely discounted in the combination. Of the remaining soft jets, all wrong combinations would only contribute in smearing the distribution for $M_{\gamma _1\gamma _2 j}$. We therefore propose to neutralize the smearing effects by averaging over all such soft jets (with $p_T>40$ GeV) in the invariant mass reconstruction and neglecting the first two leading jets for BP1. For BP2, the jets coming from the decay of VLQ to $s_h j$ is equally hard as the ones that come from the decay of $s_h$ themselves. Therefore, for BP2, the averaging is done including all jets with $p_T >40$ GeV. We plot their normalized distribution after averaging for both signal and background in Fig. 6. As the diphoton coming from the $s_h$ decay marks a kinematic edge in the distribution, this can be clearly seen to happen at the mass value of $s_h$ at 600 GeV for both BP1 and BP2 which is absent for the invariant mass distribution in the 3j hadronic channel. In addition, a much cleaner and distinct peak can be observed for the VLQ mass for BP2. In the case of BP1, as the VLQ mass at 640 GeV is quite close to the scalar $s_h$ mass of 600 GeV, resolving the VLQ (although visible) mass peak from the sharp kinematic edge is difficult. However, for a larger mass gap the peak should be distinctly identifiable as in BP2.

Fig. 7

The average of invariant mass distributions of the leading two photons with a jet for BP2

Finally, to impress upon the fact that the VLQ mass can be clearly reconstructed through the modified invariant mass variable proposed above, we show the distribution without any normalization in Fig. 7 overlaying the signal for BP2 over the SM background. It clearly shows the VLQ mass peak over the background. Thus, we find that both $s_h$ and $xd_1$ can be reconstructed clearly to determine their masses for the channel we study. As BP2 pertains to a VLQ mass of 850 GeV, we conclude that a TeV mass VLQ with such non-standard decay modes, possible for BSM scenarios which have very little mixing with the SM sector, can be observed and its mass parameters determined at the LHC with a few 100 $\hbox {fb}^{-1}$ of integrated luminosity.

4 Conclusion

In this work we have considered an $E_6$ motivated extension of the SM where the larger symmetry groups are broken at a very high scale and a residual U(1) gauge symmetry is the only remaining symmetry beyond the unbroken SM gauge symmetry. This additional U(1) then gets broken at the TeV scale through new SM singlet scalars giving rise to a TeV-scale particle spectrum with three generations of vector-like quarks and leptons and several neutral scalars. The vector-like quarks in the model have non-standard decay modes and decay into an ordinary light quark and a SM singlet scalar. Further the scalar decays either to two photons or two gluons. The current experimental limits for VLQ which do not decay directly to the SM particles are very weak and therefore allow their mass to be as light as 500 GeV. We analyzed the events from such VLQ production at the LHC with $\sqrt{s}=13$ TeV in the $2\gamma +\ge 2j$ final states and present a search strategy for observing its signals. We also studied how to reconstruct the masses for both the scalar and the VLQ through a modified construction of the invariant mass variable using the $\gamma \gamma j$ sub-system. We saw that the mass of the scalar can be reconstructed from the invariant mass distribution of the two leading photons. With the upcoming high luminosity data at the LHC, the new signal for the VLQ, proposed in this work, could provide to be an interesting channel to search for new physics beyond the SM.