Vector-like top/bottom-quark partners and Higgs physics at the LHC

Abstract

Vector-like quarks (VLQ) that are partners of the heavy top and bottom quarks are predicted in many extensions of the standard model (SM). We explore the possibility that these states could explain not only the long-standing anomaly in the forward–backward asymmetry in b-quark production at LEP, $A_\mathrm{FB}^b$, but also the more recent $\sim$2$\sigma$ deviation of the cross section for the associated Higgs production with top-quark pairs at the LHC, $\sigma (pp\rightarrow t\bar{t} H)$. Introducing three illustrative models for VLQs with different representations under the SM gauge group, we show that the two anomalies can be resolved while satisfying all other theoretical and experimental constraints. In this case, the three different models predict VLQ states in the 1–2 TeV mass range that can be soon probed at the LHC. In a second step, we discuss the sensitivity on the VLQ masses and couplings that could be obtained by means of a percent level accuracy in the measurement of ratios of partial Higgs decay widths, in particular $\Gamma (H \rightarrow \gamma \gamma )/\Gamma (H \rightarrow ZZ^*)$ and $\Gamma (H \rightarrow b\bar{b})/\Gamma (H \rightarrow WW^*)$. We show that top and bottom VL partners with masses up to $\sim$5 TeV and exotic VLQs with masses in the 10 TeV range can be probed at the high-luminosity LHC.

1 Introduction

Many extensions of the standard model (SM) of particle physics, including some that address the gauge hierarchy problem, predict the existence of additional color-triplet states with vector-like gauge couplings. Vector-like quarks (VLQs) arise, for instance, as Kaluza–Klein excitations in warped extra-dimension scenarios [1, 2] (in particular the version with SM fields in the bulk generating the fermion mass hierarchy, see for example [3–8]), excited resonances in the framework of composite models [9, 10], partners of the top quark in the little Higgs context [11, 12] and as additional states in the extended group representations of grand unified theories [13, 14]. As their masses are expected to be in the vicinity of the TeV scale, these particles are accessible at the Large Hadron Collider (LHC) and their search is therefore of prime importance. For phenomenological analyses concerning VLQs, see [15–39] (for more specific scenarios involving VLQs, see, for example, [40–48]).

At the LHC, direct experimental searches have imposed the model independent bound $m_\mathrm{VLQ} \gtrsim 800$ GeV [49–51] on VLQ masses from pair-production through strong interactions, almost independently of the electric charge. There exist also indirect constraints on the masses and couplings of these particles from electroweak (EW) precision tests as they enter the radiative corrections to EW precision observables such as the so-called oblique corrections that affect the W-boson mass $M_W$ and the effective mixing angle $\sin ^2\theta _W$ at high orders [52, 53]. In addition, third generation VLQs alter the properties of the heavy top and bottom quarks through fermion mixing and strong constraints can be e.g. obtained from the Z-boson decay into bottom quarks, $Z\rightarrow b\bar{b}$, as measured at the LEP $e^+ e^-$ collider at energies close to the Z-resonance [54, 55]. In the latter case, VLQs are (together with Kaluza–Klein excitations of electroweak gauge bosons [56]) among the very few possibilities that allow one to solve the long-standing puzzle of the forward-backward asymmetry $A_\mathrm{FB}^{b}$ whose measured value differs by $\sim {2.5} \sigma$ from the SM expectation [57].

Indirect constraints on VLQs also come from the data collected on the 125 GeV Higgs particle that has been observed at the LHC [61–63]. First, these new quarks contribute to the loop-induced Higgs couplings to pairs of gluons and photons, either through their additional exchange in the triangular loops or when altering the important top quark loop contribution by mixing [65–69]. The Higgs decay channels in the various final states detected so far by the ATLAS and CMS collaborations, namely the $H \rightarrow \gamma \gamma ,ZZ, WW$ and eventually $\tau ^+\tau ^-$ final states with the Higgs state dominantly produced in the gluon fusion mechanism $gg\rightarrow H$, set strong limits on the VLQ masses and couplings [61–63]. The sensitivity in these leading Higgs production channels, supplemented by the one in the Higgs-strahlung process $q\bar{q} \rightarrow VH$ with the $V=W,Z$ boson decaying leptonically and the Higgs state decaying into $H\rightarrow b\bar{b}$ final states, will significantly improve at the upgraded LHC with higher center of mass energies and integrated luminosities.

At a later LHC stage, a very efficient indirect probe of VLQ effects would come from associated Higgs production with top-quark pairs, $pp\rightarrow t \bar{t}H$, through a modification of the top-quark Yukawa coupling $y_t$, as the cross section is directly proportional to $y_t^2$. In fact, the combination of the data collected so far by the ATLAS and CMS collaborations in this channel displays a $\sim \! 2\sigma$ deviation from the SM expectation [63] although the sensitivity is still rather low (the deviation is close to $\sim \! 1\sigma$ in the ATLAS data and is much larger, being at the $\sim \! 2.1\sigma$ level, in the case of CMS [61, 62]). This excess in the production rate would correspond to an enhancement of the top-quark Yukawa coupling $y_t$ by a factor $\sim \! 1.4$.¹ Although it is rather premature, it is tempting to attribute this excess to the indirect presence of VLQs and this should soon be confirmed or infirmed.

In this paper, we analyze the sensitivity of present and future LHC Higgs data to the vector-like partners of the heavy top and bottom quarks. We adopt an effective approach and consider several VLQ representations under the SM gauge symmetry, so that the obtained scenarios can be embedded into various realistic high-energy frameworks. We first explore the possibility that some VLQs modify the Yukawa couplings of the heavy top and/or bottom quarks through fermion mixing and discuss the impact of this mixing on electroweak observables including those in $Z\rightarrow b\bar{b}$ decays. We also analyze the constraints that can be obtained from the LHC data on the observed Higgs particle, in particular those from the measured loop-induced Higgs couplings to gluons and photons as well as from the rates in the Higgs-strahlung production process followed by the decay $H\rightarrow b\bar{b}$.

As a main outcome of our study, we provide a natural and simultaneous explanation of the two possible deviations in heavy quark observables from SM expectations: the $pp\rightarrow t\bar{t} H$ cross section at the LHC and the $A_\mathrm{FB}^b$ asymmetry at LEP. For the production rate $\sigma (pp\rightarrow t\bar{t}H)$, the increase of the top Yukawa coupling that is necessary to explain the $\sim \! 2 \sigma$ excess has to be compensated by a destructive interference between the top and the VLQ loop contributions to the $gg\rightarrow H$ production and $H\rightarrow \gamma \gamma$ decay rates.² Such an interpretation of the anomaly in $\sigma ( pp\rightarrow t\bar{t}H)$ predicts VLQs with masses in the range 1–1.5 TeV, which should thus be directly produced at the next LHC runs.

Finally, we show that VLQs with masses up to $\sim$10 TeV can be probed by measuring precisely the ratios of the $H \rightarrow \gamma \gamma$ to $H \rightarrow ZZ^*$ and $H \rightarrow b \bar{b}$ to $H \rightarrow WW^*$ production times decay rates [70, 71], which are free of the large theoretical ambiguities that affect the absolute rates or the signal strengths [72–74] and which could be determined with an accuracy at the percent level at the high-luminosity LHC option [75–77].

The paper is structured as follows. In the next section, we describe three models which lead to VLQs that could allow for an enhancement of the top-quark Yukawa coupling and for a resolution of the $A_\mathrm{FB}^b$ puzzle. In Sect. 3, we summarize the presently available constraints that can be set on VLQS, in particular from high-precision electroweak and the LHC Higgs data. We then present in Sect. 4 our numerical results for each studied model and delineate the allowed parameter space for the masses and couplings of VLQs that accommodates the anomalies in $\sigma (pp\rightarrow t\bar{t} H)$ and $A_\mathrm{FB}^b$. Finally, in Sect. 5, we discuss the sensitivity to VLQs that can be achieved at the high-luminosity LHC through precision measurements of Higgs decay ratios. A brief conclusion is given in Sect. 6.

2 The theoretical set-up

In this section, we discuss the simplest models that include extra vector-like quarks and start by analyzing those which could accommodate the two possible anomalies in the heavy quark sector, namely an increase of the $pp\rightarrow t\bar{t}H$ production cross section and a deviation of the $A_\mathrm{FB}^b$ asymmetry from the SM expectation. In scenarios that lead to modifications of the top quark Yukawa coupling, defined in the mass basis as $y^\mathrm{SM}_t = m_t / v$ (when neglecting the three SM generation mixing with respect to the top-VLQ mixing), $m_t = 174 \pm 1$ GeV being the measured top-quark mass [57] and $v = v' \sqrt{2} \simeq 246$ GeV the Higgs vacuum expectation value, the VLQ responsible for such modifications will be denoted as a top partner $t'$ since it should have the same electric charge in order to mix with the top quark.

It turns out that the simplest SM extension with a unique $t'$ quark leads to a reduction of the top Yukawa coupling with respect to the SM value. This conclusion holds for a $t'$ embedded in a singlet, a doublet or a triplet under the $\mathrm{SU(2)_L}$ group, because the mass matrix in the $(t,t')$ field basis has the same texture in each of the three cases and generates identical mixing angles. The embedding of a single $t'$ component into a quadruplet or higher $\mathrm{SU(2)_L}$ multiplets forbids to have gauge invariant Yukawa couplings for the extra $t'$ and, hence, to induce t–$t'$ mixing. Consequently, one should include at least two extra top partners. The embedding of vector-like $t',t''$ quarks in two $\mathrm{SU(2)_L}$ singlets would lead to a mass matrix in the $(t,t',t'')$ field basis of the type (from now on, we denote by “Y” the interaction basis couplings and by “y” the mass basis couplings)

$$\begin{aligned} \mathcal {M}_t = \left( \begin{array}{ccc} Y_{t_1} v' &{} Y_{t_2} v' &{} Y_{t_3} v' \\ 0 &{} m_1 &{} 0 \\ 0 &{} 0 &{} m_2 \end{array}\right) , \end{aligned}$$

(1)

which turns out to have an insufficient number of free parameters to increase $y_t$ without significantly altering the measured $m_t$ value. The same holds for two extra isodoublets, for which the mass matrix is simply the transpose of $\mathcal {M}_t$. Therefore, in order to increase $y_t$, the minimal top sector (where we only consider the least possible number of $n$-plets with $n \le 3$) should include one $t'$ embedded in an $\mathrm {SU(2)_L}$ doublet and one $\mathrm {SU(2)_L}$ singlet, $t''$. In this paper, we consider only these minimal scenarios for the heavy quark sectors with the least possible number of $n$-plets with $n \le 3$.

In the case of the forward–backward asymmetry $A_\mathrm{FB}^b$, one can use similar arguments to construct a minimal sector. The main goal is to reduce the $A_\mathrm{FB}^b$ tension with data through tree-level changes of the $Z b\bar{b}$ couplings, induced by the mixing of the SM b-quark with its VLQ partners (see next section). However, one should keep the ratio $R_b \equiv {\Gamma (Z \rightarrow b\bar{b})}/{\Gamma (Z \rightarrow \mathrm {hadrons})}$ in agreement at the $\lesssim 1 \sigma$ level with its SM value when the tree-level $Z b\bar{b}$ coupling constants $g_{b_L}$ and $g_{b_R}$ are modified. This problem has been studied previously and a possible solution is to increase $g_{b_R}$ by $\sim \! 30\,\%$ and to decrease the absolute value of $g_{b_L}$ by $\sim \! 1\,\%$ with respect to their SM values [56, 78].

The requirement of such a large increase in the right-handed component of the $Zb\bar{b}$ coupling $g_{b_R}$ gives an idea on the minimal bottom-quark sector that is required. In the interaction basis, the coupling matrix of the Z-boson to the b-quark and its VL partners has the diagonal form

$$\begin{aligned} G_{L/R}^b= & {} \mathrm {diag}\bigg (I^{(1)}_{3L/R} + \frac{1}{3} \sin ^2 \theta _W, \, I^{(2)}_{3L/R} + \frac{1}{3} \sin ^2 \theta _W,\nonumber \\&\quad I^{(3)}_{3L/R} + \frac{1}{3} \sin ^2 \theta _W, \ldots \bigg ), \end{aligned}$$

(2)

where $I^{(1)}_{3R} = 0$ and $I^{(1)}_{3L} = -\frac{1}{2}$ are the SM $b_L$ and $b_R$ isospin projections and $I^{(2,3)}_{3L/R}$ stand for the first and second VL left-handed/right-handed $b'$’s isospin projections. Rotating to the mass basis by a unitary transformation $U_R^b$, one finds that, for the characteristic case of two $b'$ states, the $Z b_R\bar{b}_R$ coupling becomes

$$\begin{aligned} G_{R,11}^b\equiv & {} \tilde{g}_{b_R} = \sum _{i=1}^{3} I^{(i)}_{3R} \left( U_{R,1i}^b\right) ^2 + \frac{1}{3} \sin ^2 \theta _W\nonumber \\\equiv & {} I^{(1)}_{3R, \mathrm eff} + \frac{1}{3} \sin ^2 \theta _W, \end{aligned}$$

(3)

where $I^{(1)}_{3R, \mathrm eff}$ is the “effective isospin” of the SM bottom quark after mixing with its VLQ partners. Thus, after b–$b'$ mixing, the change in $g_{b_R}$ is equal to $I^{(1)}_{3R,\mathrm eff}$, since $I^{(1)}_{3R} = 0$. As unitarity implies $\sum _{i=1}^{3} (U_{R,1i}^b)^2 = 1$, one concludes that the effective isospin of the SM $b$ quark is actually a weighted mean of the isospins of all the bottom-like quarks present in the model. Since the measured values of $A_\mathrm{FB}^b$ and $R_b$ point towards $I^{(1)}_{3R, \mathrm eff} > 0$, the minimal model should contain one bottom-like VLQ with positive isospin and none with negative isospin, which from the start excludes a $(t',b')$ doublet. (Less minimal models could contain additional $b'$ quarks with negative isospins but non-significant mixings with the SM $b_R$ field, i.e. $U_{R,1i}^b \ll 1$.)

Therefore, in the sense of the minimality mentioned above, experimental constraints in the bottom sector favor a $b'$ VLQ embedded with a $-\frac{4}{3}$ electric charge VLQ, $q_{4/3}$, in a $-\frac{5}{6}$ hypercharge isodoublet,

$$\begin{aligned} B_{L,R} = \left( \begin{array}{c} b' \\ q_{4/3} \end{array} \right) ^{Y=-5/6}_{L,R}, \end{aligned}$$

(4)

with the addition of a singlet $b''$, which guarantees that there are enough parameters to produce a significant deviation of the couplings to the $Z$ boson [79]. The electric charge of the multiplet components is fixed by the relation $Q = Y + I_3$ coming from the assumption that the symmetry breaking occurs as in the SM. The hypercharge is fixed by the gauge symmetry itself, which imposes the same Y value for the components of a given multiplet. In addition, as the bottom sector measurements disfavor a $(t',b')$ doublet and the minimal top sector imposes a $t'$ embedded in a doublet, one concludes that the $t'$ should pair up with an exotic electric charge $+\frac{5}{3}$ VLQ, $q_{5/3}$, in an $\mathrm SU(2)_L$ doublet,

$$\begin{aligned} T_{L,R} = \left( \begin{array}{c} q_{5/3} \\ t' \end{array} \right) ^{Y=7/6}_{L,R}. \end{aligned}$$

(5)

Along these lines, one can construct a minimal VLQ model, which we denote here as model A, which addresses simultaneously the excess of the $pp\rightarrow t \bar{t}H$ cross section at the LHC and the anomaly in the $A_\mathrm{FB}^b$ asymmetry as measured at LEP. Besides the SM fields, model A will have the following content:

$$\begin{aligned} \mathrm{Model}~\mathbf {A}: T_{L,R}, \; B_{L,R}, \; t''_{L,R} \quad \mathrm {and} \; b''_{L,R}, \end{aligned}$$

(6)

where $B_{L,R}$ and $T_{L,R}$ are the two isodoublets defined in Eqs. (4) and (5), respectively, whereas $b''_{L,R}$ and $t''_{L,R}$ are two isosinglets. Denoting the SM left-handed (t, b) doublet as $Q_L$, the most general Lagrangian containing all possible terms invariant under the SM $\mathrm{SU(3)_C \! \times \! SU(2)_L \! \times \! U(1)_Y}$ gauge symmetry reads

$$\begin{aligned} \mathcal {L}&=Y_{t_1} \, \overline{Q}_L \tilde{H} \, t_R \, + \, Y_{t_2} \, \overline{Q}_L \tilde{H} \, t''_R + Y_{t_3} \, \overline{T}_L H \, t_R \nonumber \\&\quad + Y_{t_4} \, \overline{T}_L H \, t''_R+ Y_{t_5} \, \overline{T}_R H \, t''_L +Y_{b_1} \, \overline{Q}_L H \, b_R \nonumber \\&\quad + Y_{b_2} \, \overline{Q}_L H \, b''_R+ Y_{b_3} \, \overline{B}_L \tilde{H} \, b_R+ Y_{b_4} \, \overline{B}_L \tilde{H} \, b''_R\nonumber \\&\quad + Y_{b_5} \, \overline{B}_R \tilde{H} \, b''_L+m_1 \, \overline{T}_L T_R + m_2 \, \overline{t}''_L t''_R \nonumber \\&\quad +m_3 \, \overline{B}_L B_R+ m_4 \, \overline{b}''_L b''_R + \mathrm {H.c.}, \end{aligned}$$

(7)

where $H = \left( \begin{array}{c} H^+ \\ H^0 \end{array} \right)$ represents the SM Higgs doublet, $\tilde{H}=i \sigma _2 H^*$ its charge conjugate, L / R the left and right fermion chiralities, the Y’s dimensionless Yukawa coupling constants and m’s the masses of the various VLQs.

Without loss of generality, the coefficients of the $\overline{t}''_L t_R$ and $\overline{t}_L t'_R$ terms can be rotated away [80]. The Yukawa couplings for the first two generations of fermions are omitted in the Lagrangian of Eq. (7) as their mixings with the top partners $t',t''$ are expected to be much smaller than the t–$t'$ and t–$t''$ mixings as a consequence of the larger mass differences. Since the CKM angles [57] are typically small, the first two up-quark flavors naturally decouple from the top quark. A similar discussion holds for the down-type quark sector and the $b'$, $b''$ components.³

The top Yukawa couplings and mass terms generated after symmetry breaking by the Lagrangian of Eq. (7) can be synthesized, respectively, in the $h \bar{\psi }_L^t \mathcal{C}_t \psi _R^t$ and $\bar{\psi }_L^t \mathcal{M}_t \psi _R^t$ terms (the “t” superscript stands for “top”, while the T superscript stands for matrix transposition). Within the interaction basis defined by $\psi ^t=(t,t',t'')^T$, the coupling and mass matrices read

$$\begin{aligned} \mathcal{C}_t = \frac{1}{\sqrt{2}} \ \left( \begin{array}{ccc} Y_{t_1} &{} 0 &{} Y_{t_2} \\ Y_{t_3} &{} 0 &{} Y_{t_4} \\ 0 &{} Y_{t_5} &{} 0 \end{array} \right) ,\quad \mathcal{M}_t = \left( \begin{array}{ccc} v'Y_{t_1} &{} 0 &{} v' Y_{t_2} \\ v' Y_{t_3} &{} m_1 &{} v' Y_{t_4} \\ 0 &{} v' Y_{t5} &{} m_2 \end{array} \right) . \end{aligned}$$

(8)

In the mass basis (“m” superscript), one has $\mathcal{C}_t^m\!=\!U_{L}^t \mathcal{C}_t (U_{R}^t)^\dagger$, where the unitary matrices $U_{L/R}^t$ are obtained by bi-diagonalizing the model dependent mass matrix, $U_L^t \mathcal{M}_t (U_R^t)^\dagger =\mathrm{diag}(m_{t_1},m_{t_2},m_{t_3})$. The argument stays the same for the b-quark sector, but with the replacements $t\rightarrow b$, $m_1 \rightarrow m_3$ and $m_2 \rightarrow m_4$. As for the $\frac{5}{3}$ and $-\frac{4}{3}$ charged exotic partners, their masses are given by $|m_1|$ and $|m_3|$, respectively. The mass eigenstates obtained in the mass basis are ordered by increasing absolute value and thus, for example, the observed top quark (after mixing) will be represented by $t_1$, while the lightest bottom-like VLQ will be denoted by $b_2$.

We will show later that indeed, there is a region in the parameter space of this minimal model where all the LEP and LHC constraints, as well as the constraints from the oblique corrections that affect the W / Z propagators, are satisfied. However, for the sake of completeness, we will also consider two other models that respect too the requirement of minimality and pass the constraints mentioned above. The two additional models contain, besides the SM fields, the VLQ multiplets enlisted below:

$$\begin{aligned} \mathrm{Model}~\mathbf {B}: T_{L,R}, \; B_{L,R}, \; X_{L,R} = \left( \begin{array}{c} t'' \\ b'' \\ q'_{4/3} \end{array} \right) ^{Y=-1/3}_{L,R}\quad \mathrm {and} \quad t'''_{L,R}.\nonumber \\ \end{aligned}$$

(9)

This is simply a copy of the minimal model A with the replacement $b'' \rightarrow X$, with the top-like singlet from model A being renamed into $t'''$. The triplet is chosen such that the isospin of $b''$ is equal to 0, which, together with $b'$ having a positive isospin, solves the $A_\mathrm{FB}^b$ discrepancy. Also, with the choice of this triplet, this model has the same number of parameters as model A, namely 14. We have

$$\begin{aligned} \mathrm{Model~}\mathbf {C}: T_{L,R}, \; B_{L,R}, \; Z_{L,R} = \left( \begin{array}{c} q_{8/3} \\ q'_{5/3} \\ t'' \end{array} \right) ^{Y=5/3}_{L,R}, \; b''_{L,R} \quad \mathrm {and} \; t'''_{L,R}.\nonumber \\ \end{aligned}$$

(10)

Just as in the previous model, the top-like singlet gets the most primes, becoming $t'''$. In both models, the B and T VLQ doublets are the ones defined earlier in Eqs. (4) and (5).

We close this general discussion by presenting the Lagrangians and the mass matrices of the additional models B and C. We denote the interaction basis vectors as $\psi ^q = \left( q,q',q'',\ldots \right) ^T$, where $q$ stands for the quark type, namely $b$, $t$, $q_{4/3}$ and $q_{5/3}$, while $(\ldots )^T$ stands for the matrix transpose operation. The Yukawa coupling matrices will not be written, since they are obtained in a straightforward manner by differentiating the corresponding mass matrices with respect to the Higgs vev, v (recall that $v = v' \sqrt{2}$).

Model B: the corresponding Lagrangian is given by

$$\begin{aligned} \mathcal {L}&= Y_{t_1} \, \overline{Q}_L \tilde{H} \, t_R + Y_{t_2} \, \overline{Q}_L H \, X_R + Y_{t_3} \, \overline{Q}_L \tilde{H} \, t'''_R \nonumber \\&\quad + Y_{t_4} \, \overline{T}_L H \, t_R+ Y_{t_5} \, \overline{T}_L H \, t'''_R + Y_{t_6} \, \overline{T}_R H \, t'''_L\nonumber \\&\quad + Y_{b_1} \, \overline{Q}_L H \, b_R + Y_{b_2} \, \overline{B}_L \tilde{H} \, b_R\nonumber \\&\quad + Y_{b_3} \, \overline{B}_L \tilde{H} \, X_R + Y_{b_4} \, \overline{B}_R \tilde{H} \, X_L + m_1 \, \overline{T}_L T_R + m_2 \, \overline{X}_L X_R \nonumber \\&\quad + m_3 \, \overline{t}'''_L t'''_R+ m_4 \, \overline{B}_L B_R + \mathrm {H.c.}, \end{aligned}$$

(11)

For the top, bottom and $-\frac{4}{3}$ electric charge quarks, the mass matrices are given by

$$\begin{aligned}&\mathcal {M}_t = \left( \begin{array}{cccc} v'Y_{t_1} &{} 0 &{} v'Y_{t_2} &{} v'Y_{t_3} \\ v'Y_{t_4} &{} m_1 &{} 0 &{} v'Y_{t_5} \\ 0 &{} 0 &{} m_2 &{} 0 \\ 0 &{} v'Y_{t_6} &{} 0 &{} m_3 \end{array} \right) , \nonumber \\&\mathcal {M}_b = \left( \begin{array}{ccc} v'Y_{b_1} &{} 0 &{} \frac{v' Y_{t_2}}{\sqrt{2}} \\ v'Y_{b_2} &{} m_4 &{} \frac{v'Y_{b_3}}{\sqrt{2}} \\ 0 &{}\frac{v'Y_{b_4}}{\sqrt{2}} &{} m_2 \end{array} \right) , \; \nonumber \\&\mathcal {M}_{4/3} = \left( \! \begin{array}{cc} m_4 &{} v'Y_{b_3} \\ v'Y_{b_4} &{} m_2 \end{array} \! \right) . \end{aligned}$$

(12)

Additionally, the physical mass of $q_{5/3}$ is given by $|m1|$. Note that, in the bottom-quark mass matrix from above, the $Y_{t_2}, Y_{b_3}$ and $Y_{b_4}$ terms are divided by $\sqrt{2}$. The extra $1/\sqrt{2}$’s are just Clebsch–Gordan factors arising from the direct product of the Higgs doublet with a VL doublet into a triplet, i.e. the $\mathbf {3}$-representation from the group product decomposition $\mathbf {2} \otimes \mathbf {2} = \mathbf {3} \oplus \mathbf {1}$.

Model C: the Lagrangian is given by

$$\begin{aligned} \mathcal {L}&=Y_{t_1} \, \overline{Q}_L \tilde{H} \, t_R + Y_{t_2} \, \overline{Q}_L \tilde{H} \, t'''_R + Y_{t_3} \, \overline{T}_L H \, t_R + Y_{t_4} \, \overline{T}_L \tilde{H} \, Z_R \nonumber \\&\quad + Y_{t_5} \, \overline{T}_L H \, t'''_R + Y_{t_6} \, \overline{Z}_L H \, T_R + Y_{t_7} \, \overline{T}_R H \, t'''_L + Y_{b_1} \, \overline{Q}_L H \, b_R\nonumber \\&\quad + Y_{b_2} \, \overline{Q}_L H \, b''_R+ Y_{b_3} \, \overline{B}_L \tilde{H} \, b_R + Y_{b_4} \, \overline{B}_L \tilde{H} \, b''_R\nonumber \\&\quad + Y_{b_5} \, \overline{B}_R \tilde{H} \, b''_L + m_1 \, \overline{T}_L T_R \nonumber \\&\quad + m_2 \, \overline{Z}_L Z_R+ m_3 \, \overline{t}'''_L t'''_R + m_4 \, \overline{B}_L B_R + m_5 \, \overline{b}''_L b''_R + \mathrm {H.c.}, \end{aligned}$$

(13)

and the mass matrices for the t, b, and $\frac{5}{3}$ electric charge quarks are given by

$$\begin{aligned}&\mathcal {M}_t = \left( \begin{array}{cccc} v'Y_{t_1} &{} 0 &{} 0 &{} v'Y_{t_2} \\ v'Y_{t_3} &{} m_1 &{} v'Y_{t_4} &{} v'Y_{t_5} \\ 0 &{} v'Y_{t_6} &{} m_2 &{} 0 \\ 0 &{} v'Y_{t_7} &{} 0 &{} m_3 \end{array} \right) , \nonumber \\&\mathcal {M}_b = \left( \begin{array}{ccc} v'Y_{b_1} &{} 0 &{} v'Y_{b_2} \\ v'Y_{b_3} &{} m_4 &{} v'Y_{b_4} \\ 0 &{} v'Y_{b_5} &{} m_5 \end{array} \right) , \nonumber \\&\mathcal {M}_{5/3} = \left( \begin{array}{cc} m_1 &{} \frac{v' Y_{t_4}}{\sqrt{2}} \\ \frac{v' Y_{t_6}}{\sqrt{2}} &{} m_2 \end{array} \right) . \end{aligned}$$

(14)

The novelty of this model is the appearance of an electric charge $\frac{8}{3}$ exotic quark, $q_{8/3}$, whose mass is given by $|m_2|$. Also, the mass of $q_{4/3}$ is given by $|m_4|$.

3 Present constraints on the VLQ properties

3.1 Bounds from the LHC Higgs data

The first set of constraints that we consider is due to Higgs production and detection at the LHC; for a review of the relevant processes see e.g. Ref. [81]. The data collected by the ATLAS and CMS collaborations at $7+8$ TeV c.m. energies in the main search channels, namely the $H\rightarrow \gamma \gamma , ZZ, WW, \tau \tau$ detection modes with the Higgs boson produced in the gluon (ggF) and in the vector boson (VBF) fusion channels plus the $H \rightarrow b\bar{b}$ decay mode with the Higgs produced in the $q\bar{q} \rightarrow VH$ mode (VH) with $V=W,Z$, seem to be in good agreement with the SM expectations [61–63]. One can thus use the signal strengths $\mu _{XX}$ in these Higgs detection channels, defined as the measured cross section times the decay branching ratio relative to the SM prediction,

$$\begin{aligned} \mu _{XX}= \frac{\sigma ( pp\rightarrow H)}{\sigma ( pp\rightarrow H)_\mathrm{SM}} \times \frac{\mathrm{BR}(H\rightarrow XX)}{\mathrm{BR}(H\rightarrow XX)_\mathrm{SM}}, \end{aligned}$$

(15)

to constrain possible effects of extra vector-like top and bottom partners which would impact several of them.

The cross section for the gluon fusion mechanism ggF is by far the dominant Higgs production process at the LHC as it provides $\sim$85 % of the total Higgs sample before kinematical cuts are applied. In the SM, the process is mediated by triangular top and (to a lesser extent) bottom-quark loops. VLQs that are top and bottom partners would affect the ggF production rate either through mixing, i.e. by modifying the t, b loop contributions, or their exchange in the loop (the various quark contributions to the loop-induced Hgg coupling are summarized in the appendix). The virtual impact of VLQs in the Hgg vertex can be probed essentially through the signal strength in the $H\rightarrow ZZ^* \rightarrow 4\ell ^\pm$ channel that is among the most precisely measured ones (we refrain here from adding the information from the $H \rightarrow WW^* \rightarrow 2\ell 2\nu$ search channel that is affected by larger theoretical and experimental uncertainties). Averaging the most recent ATLAS and CMS measurements [61–63], one obtains⁴ [61, 62]

$$\begin{aligned} \mu _{ZZ}^\mathrm{(comb)} = 1.17^{+0.23}_{-0.22} . \end{aligned}$$

(16)

The loop-induced $H\rightarrow \gamma \gamma$ decay mode bears many similarities with the ggF process. It is mediated by top and bottom quark triangular loops but has also contributions from the W boson which, in fact, is dominating and interferes destructively with that of the heavy quarks. Again, additional contributions come from VLQs, in particular through their exchange in the $H\gamma \gamma$ vertex (the impact of VLQs in this channel is also summarized in the appendix). Given their smaller electric charge, VLQ bottom-quark partners barely contribute to the vertex but exotic VLQs with higher electric charge, e.g. $+\frac{5}{3}$ or $-\frac{4}{3}$, could more significantly affect the loop [67]. Present ATLAS and CMS data [61, 62], when combined, give the even stronger constraint

$$\begin{aligned} \mu _{\gamma \gamma }^{\mathrm {(comb)}} = 1.14 \pm 0.18. \end{aligned}$$

(17)

Additional bottom-like VLQ partners would alter the $Hb\bar{b}$ coupling in addition to the $Z b \bar{b}$ vertex. Consequently, one should also enforce the constraint from the Higgs-strahlung process with the Higgs boson observed in the $H\rightarrow b\bar{b}$ signature. Combining the ATLAS and CMS results [62, 84], one obtains for this channel⁵

$$\begin{aligned} \mu _{bb}^\mathrm{(comb)} = 0.69 \pm 0.29. \end{aligned}$$

(18)

Note that here the production cross section in the VH process is not altered at tree level by the presence of VLQ and only the $H\rightarrow b\bar{b}$ branching ratio is affected. In fact, this branching ratio, $\sim$60 %, is the dominant one [85]. It controls the total decay width and therefore enters in all the other Higgs branching ratios and hence all signal strengths. We will thus simultaneously include the various effects and impose the three constraints from $\mu _{ZZ}, \mu _{\gamma \gamma }$ and $\mu _{bb}$ at the same time, ignoring the other signal strengths that are less stringently constrained [61–63].

Finally, we will also consider the signal in the associated $pp\rightarrow t\bar{t}H$ production channel for which the combined ATLAS and CMS measurement [61–63]

$$\begin{aligned} \mu _{ttH}^\mathrm{(comb)} = 2.23^{+0.64}_{-0.61} \end{aligned}$$

(19)

exhibits a $\sim$2$\sigma$ excess compared to the SM value, which is very tempting to attribute to new physics. The experimental value for the $\mu _{ttH}$ signal strength assumes SM Higgs decay rates, a feature that is consistent as the decays modified by VLQs such as the $H \rightarrow b \bar{b}$ and $H\rightarrow \gamma \gamma$ modes will be separately tested here to be close to their SM values.

In our discussion, this deviation will be attributed to an enhancement of the top-quark Yukawa coupling as a result of mixing with a VLQ partner. However, because the non-SM-like $y_t$ coupling would also affect the top-quark contributions to the Hgg and $H\gamma \gamma$ vertices, one could compensate the $y_t$ enhancement by another (negatively interfering) contribution due to VLQ exchanges in the loops as these effective couplings seem to be in agreement with the SM prediction.

3.2 Constraints from high-precision tests

There are also indirect constraints on VLQs from high-precision electroweak data. First, for the third generation quark sector, there are tree-level corrections induced by the t–$t'$ or b–$b'$ mixings directly on the t or b vertices but, because of the heaviness of $t'$ states, the value for the CKM matrix element $V_{tb}$ [57] including quark mixing is expected to be SM-like. In addition, there are radiative corrections to the gauge boson vacuum polarization functions induced by the exchange of VLQs [86, 87]. These can be cast into the so-called “oblique” parameters S, T, and U [52] that must lie inside the $1\sigma$ regions induced by a long list of electroweak precision observables [57]. Three crucial observables, the W boson mass $M_W$, the leptonic partial width $\Gamma ( Z\rightarrow \ell \ell )$, and the longitudinal polarization and forward–backward asymmetries for leptons that give $\sin ^2\theta _W$ play a prominent role [57]. Given the fact that none of our considered models exhibit an explicit custodial symmetry, it is a non-trivial question whether they will respect these oblique parameter constraints. Also, trying to impose custodial symmetry, as in Ref. [79], would not be a valid solution, since a strong mixing between the SM quarks and the VLQs is needed to explain the $\mu _{ttH}$ enhancement. Moreover, such a symmetry would require a high number of VLQ multiplets, which goes against the idea of minimality.

We will analyze the $2 \sigma$ excursions of the correlated S and T values (with the usual assumption that $U = 0$), obtained for our models, from the experimental values of the two parameters, which are given by

$$\begin{aligned} S|_{U=0} = 0.06 \pm 0.09\quad \mathrm{and}\quad T|_{U=0} = 0.10 \pm 0.07, \end{aligned}$$

(20)

with a correlation coefficient of $0.91$ [88]. We find that the theoretical prediction for the $S$ parameter typically does not deviate too much from its central value, while $T$ has a very high sensitivity to the addition of VLQs. Disentangling the deviations of the observable T that are due to mixing effects or to the VLQ loop contributions is rather difficult in practice. In particular, the mixing effects between (at least) three states are very cumbersome to handle; they can be treated only numerically and one then needs to resort to a scan approach as will be done in our analysis.

The other set of constraints comes from $Z\rightarrow b\bar{b}$ decays at LEP and one has for the experimental [57] and theoretical [57, 89] values of the ratio of partial widths $R_b$ and the asymmetry $A_\mathrm{FB}^b$, the following values:

$$\begin{aligned} R_b^\mathrm{(exp)} = 0.21629 \pm 0.00066&~ \mathrm{vs}~&R_b^\mathrm{(SM)} = 0.2158 \pm 0.00015\nonumber \\ \end{aligned}$$

(21)

$$\begin{aligned} A_\mathrm{FB}^\mathrm{b (exp)} = 0.0992 \pm 0.0016&~ \mathrm{vs}~&A_\mathrm{FB}^{b \,(\mathrm SM)} = 0.1029 \pm 0.0003\nonumber \\ \end{aligned}$$

(22)

As already mentioned in several instances, the models that we consider address the $A_\mathrm{FB}^b$ anomaly. They can be realized within concrete warped extra-dimensional [78] or their dual composite Higgs scenarios [90, 91]. Indeed, in model A, the VLQs could be interpreted as Kaluza–Klein excitations of SM quarks in extra-dimensional scenarios. The presence of Kaluza–Klein excitations of the bottom quark would induce b–$b'$ mixing and thus corrections to the $Zb\bar{b}$ couplings that affect $A_\mathrm{FB}^b$ and $R_b$. Furthermore, extra $t'$ modes would be simultaneously added to enhance the top-quark Yukawa coupling. These $t'$ states would then typically have a negative $\mathrm{SU(2)_L}$ isospin, as explained in Sect. 2. Such a $t'$ isospin arises in several embeddings in a $\mathrm{SU(2)_L} \times \mathrm{SU(2)_R}$ custodial symmetry gauged in the bulk which allows a protection with respect to all electroweak precision data [78, 90–92]. In other words, the extra-dimensional scenarios that comply with the S, T constraints could naturally predict an enhanced $y_t$ coupling and a smaller value for $A_\mathrm{FB}^b$, at least from the point of view of the field content and their gauge group embedding.

Note that since we are considering a unique set of VLQ fields and not a replica per generation, it means that the so-called custodians ($t', b', \ldots$) for the first two quark (and three lepton) SM generations would decouple, which can be realistic in such frameworks [78, 91]. Higgs data imply then large masses for the Kaluza–Klein excitations of gauge bosons and the Higgs sector would essentially feel only the effects of the VLQs (custodians) from the various effective A–C models.

3.3 Other constraints

Apart from the constraints coming from the LEP and LHC, one should also incorporate various constraints concerning the eigenmasses of the physical states and their couplings to the scalar Higgs field. First, one should reproduce the observed top- and bottom-quark masses. However, since we are neglecting the mixing between the three flavors and also the running from the VLQ mass down to the heavy quark pole masses $m_t$ and $m_b$, we will allow for an uncertainty for both eigenmasses. In the case of the top quark $t_1$, we require its mass to lie between 157 and 191  GeV, which represents a 10 % excursion from the measured value of $m_{t_1} \sim 174$ GeV. As for the bottom quark $b_1$, we impose for its mass a value between 3 and 5 GeV.

Indeed, as the Cabibbo–Kobayashi–Maskawa (CKM) matrix is close to the identity matrix, the simplest theoretical quark mixing configuration corresponds to having both rotation matrices for the up and down quark sectors close to identity as well, with a similar assumption for the matrices of the right-handed sector, for simplicity. Since the deviation of the CKM matrix from identity (its off-diagonal elements) is between less than 1 % and $\sim$20 % [57], one can expect deviations of order 10 % in these up and down rotation matrices. Hence, the top-quark mass could be affected by a correction of this order. Such mixing effects can induce an even larger uncertainty for the bottom quark due to its mass being closer to the light generation ones.

Besides, the running effect between the VLQ and the top-quark pole masses could be of order $\sim$10 % [58, 59], depending on the considered model. The bottom-quark mass can be even more affected due to its proximity to the QCD scale, $\Lambda _\mathrm{QCD}$.

Second, one should take into account the mass constraints coming from direct searches for VLQs at the LHC. Up to date, the most severe bounds on the VLQ masses come from the ATLAS experiment and are as follows:

• for a top-like VL partner, $m_{t_2} > 950$ GeV for $\mathrm {BR} (t_2\rightarrow Ht_1) = 1$ [50];

• for a bottom-like VLQ, $m_{b_2} > 813$ GeV for $\mathrm {BR} (b_2 \rightarrow W t_1) = 1$ [50];

• for a $+\frac{5}{3}$ charged VLQ, $m_{5/3_1}>840$ GeV for $\mathrm {BR} (q_{5/3_1}\rightarrow Wt_1) = 1$ [49];

• for a $-\frac{4}{3}$ charged VLQ, $m_{4/3_1} > 770$ GeV  for $\mathrm {BR} (q_{4/3_1} \rightarrow Wb_1) = 1$ [49].

To be conservative, we have considered for each type of VLQ the decay branching ratio values that give the most stringent lower bound on their eigenmass.

However, since it contains an electric charge $+\frac{8}{3}$ VLQ, model C needs an additional discussion. At LHC, there are no dedicated searches for such a resonance, but the authors of Ref. [60] have recast the LHC exclusion limits for an electric charge $\frac{5}{3}$ VLQ giving same-sign dilepton final states into the bound $m_{8/3} > 940$ GeV at 95 % CL. In our analysis, this is the limit that we will use.

Besides one $\frac{8}{3}$ VL quark, model C contains two $\frac{5}{3}$ VLQs, which in general are not degenerate in mass. Given this situation, one must reinterpret the mass exclusion limits on a $\frac{5}{3}$ top partner. Supposing that the heavier 5/3 charge partner decays always in the lighter $\frac{5}{3}$ VLQ (plus a Z or a Higgs boson) and knowing that $\mathrm {BR} (q_{8/3} \rightarrow q_{5/3} + W) = 1$, one can infer a lower bound on the mass of the electric charge $\frac{5}{3}$ partner. A conservative bound can be obtained by considering that $m_{8/3} \simeq m_{5/3_2} \simeq m_{5/3_1}$ (mass differences of $\mathcal {O}(m_W)$, thus negligible), which, together with the previous assumption, amounts to multiplying by a factor 3 the QCD pair-production cross section of the charge 5/3 states. This assumption gives the most conservative bound because, by minimizing the masses of the VLQs, we maximize their production cross section. Reinterpreting the search for electric charge 5/3 colored resonances from Ref. [49] in this way, we obtain, in the particular case of model C, the conservative bound $m_{5/3} \gtrsim 1$ TeV at 95 % CL.

Fig. 1

In the context of model A, the domains in the $[Y_{t_3},Y_{t_4}]$ plane of the Yukawa couplings in the top sector (left plot) and $[Y_{b_2},Y_{b_3}]$ of the Yukawa couplings in the bottom sector (right plot) in which the various experimental constraints are satisfied at the 1$\sigma$ level. The constraints discussed in the text but not displayed (such as the one from the perturbativity of the Yukawa couplings and the lower limits on the VLQ masses) are satisfied in the entire planes. The regions complying with all constraints are highlighted by the orange crosses

Finally, to make our predictions reliable at leading order in perturbation theory, we impose a perturbativity bound on the Yukawa couplings in the mass basis. Using naive dimensional analysis, we thus enforce the conservative constraint $\max ( |y_{ij}| ) < \sqrt{4\pi }$ for all four types of quarks, namely top, bottom, $q_{4/3}$ and $q_{5/3}$ states.

4 Numerical analysis

We now present our numerical results on the constraints on VLQ masses and couplings from current data. We first summarize the approximations that we use when enforcing the various constraints from the Higgs signal strengths as defined in Eq. (15) and as measured at the first run of the LHC, Eqs. (16)–(18). As discussed previously, on the production side, the additional VLQ can only alter the ggF production mechanism as it does not affect the HVV couplings that enter in the subdominant VBF and VH processes. Since the ggF process is responsible for most of the Higgs production cross section at the LHC, we assume that both in the SM and in our VLQ models, one simply has $\sigma (pp \rightarrow H) \simeq \sigma (gg \rightarrow H)$ for all signal strengths with the exception of the $H \rightarrow b\bar{b}$ decay. In the latter case, the production mode is instead the Higgs-strahlung process which should be SM-like. Moreover, we consider that only the decay $H \rightarrow b\bar{b}$, which has the largest branching ratio, is modified in the presence of the VLQs ($H \rightarrow \gamma \gamma$ is also modified, but this decay has a negligible branching ratio). Indeed, the other decay mode that involves third generation quark couplings, namely $H\! \rightarrow \! gg$, has a small branching ratio and is expected to be close to its SM value, as the vertex is tested directly via the production process. Thus, we consider that the modification of the total decay width of the Higgs boson, which enters in all signal strengths, comes only from the altered $H b\bar{b}$ vertex.

Fig. 2

The same as in Fig. 1 but in the context of model B

Considering first model A, we present in the left-hand side of Fig. 1 the constraints that we obtain in the $[Y_{t_3},Y_{t_4}]$ plane. The solid black lines and the gray lines delineate, respectively, the domains where the signal strengths $\mu _{ZZ}$ and $\mu _{\gamma \gamma }$ respect the LHC measurements given in Eqs. (16, 17), at the 1$\sigma$ level. The black dashed lines delineate the areas in which the constraints from the electroweak precision oblique parameter S and T are satisfied at $1\sigma$ and $1\sigma$, with $U=0$, while at the right of the red line, the top-quark mass is reproduced within an uncertainty of $\pm 10\,\%$ (the lower value does not appear in this frame). The regions excluded by the non-perturbativity of the Yukawa couplings or by too low VLQ masses are included but their impact in also not shown in the figure. Finally, the region in which the top-quark Yukawa coupling $y_{t_1}$ needs to be enhanced so as to explain the observed excess in the $t \bar{t} H$ production rate relative to the SM prediction is given by the blue lines: the lines for $\mu _{ttH}= 2.87$ and $\mu _{ttH}= 1.62$, which correspond respectively to the $+1\sigma$ and $-1\sigma$ deviation of the experimental value as given in Eq. (19), as well as the central value $\mu _{ttH}= 2.23$, are shown.

Turning to the bottom sector, we display in the right-hand side of Fig. 1 the regions in the $[Y_{b_2},Y_{b_3}]$ plane where the various experimental (and theoretical as we also include the perturbativity of the couplings) constraints are satisfied. Apart from imposing no more than $1 \sigma$ deviation compared to the SM for the measured values of the $A_\mathrm{FB}^b$ asymmetry (purple lines) and the $R_b$ ratio of widths (green lines), we allow for the bottom-quark mass to take values between $3$ and $5$ GeV (to account for the neglected effects of running and flavor mixing), a constraint that is not displayed in the figure as it is satisfied in the entire plane. Also not displayed, the LHC constraint on the $\mu _{bb}$ signal strength is compatible with data at the 1$\sigma$ level in the whole plane (the experimental central value given in Eq. (18) is $\sim$1$\sigma$ smaller than the expectation in the SM). This is not the case of the $\mu _{ZZ}$ and $\mu _{\gamma \gamma }$ constraints which, as in the top sector case, are depicted by the solid black and the solid gray lines respectively. Here, the additional constraints on the mass of the VL bottom-quark partners from direct LHC searches play an important role. Naturally, the constraints on the top-quark mass $m_{t_1}$ and the signal strength $\mu _{ttH}$ do not appear in the plot as they essentially depend on parameters from the top sector (likewise, the constraints from $A_\mathrm{FB}^b$ and $R_b$ do not appear on the left plot of Fig. 1 as they also do not depend on the top sector parameters).

In Fig. 1, the regions with the orange crosses are the ones that are compatible, at the 68 % confidence level (95 % CL for S and T), with all considered constraints. In the bottom sector plot, we fix the $Y_{t_3}$ and $Y_{t_4}$ interaction basis parameters at $Y_{t_3} = -1.45$ and $Y_{t_4} = 4.32$, while in the top sector plot we take $Y_{b_2}=-3.15$ and $Y_{b_3} = -1.08$. The values of the other parameters of the considered model A that appear in the Lagrangian of Eq. (7) are given by $Y_{t_1} = -0.98$, $Y_{t_2}=3.05$, $Y_{t_5} = 3.81$, $Y_{b_1} = -0.02$, $Y_{b_4} = -2.2$, $Y_{b_5} = -0.05$, for the Yukawa couplings and $m_1 = 1.77$ TeV, $m_2 = 1.61$ TeV, $m_3 = -0.85$ TeV and $m_4 = - 5.69$ TeV for the VLQ masses.

The outcome of the discussion is that indeed, there is a set of parameters that satisfies at the same time the LHC Higgs data and explains in particular the observed excess in the cross section of the $t \bar{t} H$ process, and the EW precision data accommodating in particular the observed discrepancy of the $A_\mathrm{FB}^b$ asymmetry compared to the SM value. As already mentioned, this is a rather non-trivial result. As the masses of the majority of the VLQs that result from the fit lie in the range between 1 and 2 TeV, this scenario should be soon checked at the LHC by producing directly the additional VLQ states.

The same considerations apply for models B and C and we show in Figs. 2 and 3 respectively the impact of the various constraints in the $[Y_{t_3},Y_{t_4}]$ (left plots) and $[Y_{b_2},Y_{b_{3,4}}]$ (right plots) planes. The allowed regions in which the b-quark related constraints $m_{b_1}, A_\mathrm{FB}^b, R_b, \mu _{bb}$, the t-quark related constraints $m_{t_1}, \mu _{ttH}$, the general constraints $\mu _{\gamma \gamma }, \mu _{ZZ}$, as well as the EW constraints $S \, \& \, T$ and the lower LHC bounds on the VLQ masses are satisfied at the $1\sigma$ level ($2\sigma$ for $S \, \& \, T$) are also highlighted by orange crosses. We have also enforced the perturbativity of all Yukawa couplings and the impact of this constraint is now visible in the plots: the dark colored areas are those where at least one Yukawa coupling in the mass basis is larger than $\sqrt{4\pi }$ in absolute value. Also, we imposed the direct exclusion limit on the VLQ mass $m_{b_2} > 813$ GeV [50]. One constraint is particularly important in the two models B and C, namely the $H \rightarrow \gamma \gamma$ signal strength, as both models contain VLQs with high electric charge, $-\frac{4}{3}$ and $\frac{5}{3}$, which could lead to important contributions to the $H\gamma \gamma$ vertex. Note that in the left plot of Fig. 3, only the line for the $-1\sigma$ value of the $t \bar{t} H$ rate is displayed as we have selected the areas in which the top Yukawa coupling is sufficiently enhanced to accommodate the observed excess. Moreover, one can see that in the bottom sector plot of model C (right plot of Fig. 3) there are two disjoint regions where all the phenomenological constraints mentioned above are satisfied. Although they are situated roughly symmetrically with respect to the $Y_{b_2} = 0$ line, their shapes are different. This shows that, with all the other parameters fixed, flipping the sign of $Y_{b_2}$ is of physical importance. Indeed, such a transformation changes the value of $\det \mathcal {M}_b$, which enters directly in the rate expression for the loop-induced ggF mechanism and $H \rightarrow \gamma \gamma$ decay (as described in the appendix).

Fig. 3

The same as in Fig. 1 but in the context of model C

In the two models, the values of the various parameters appearing (and defined) in Eq. (12) for model B and Eq. (14) for model C and not shown in the planes are as follows. In model B, we have $Y_{t_1} = -0.98$, $Y_{t_2} = 0.68$, $Y_{t_5} = -3.6$, $Y_{t_6} = 4.4$, $Y_{b_1} = 0.019$, $Y_{b_2} = 1.47$, $Y_{b_3} = 0.28$, $Y_{b_4} = 0.17$, $m_1 = 1.42$ TeV, $m_2 = 1.1$ TeV, $m_3 = -2.32$ TeV and $m_4 = 1.5$ TeV, with $Y_{b_2} = 1.61$ and $Y_{b_4} = 0.23$ in the top sector plot (left) plus $Y_{t_3} = -3.22$ and $Y_{t_4} = -1.21$ in the bottom sector plot (right). In model C, we have $Y_{t_1} = -1.01$, $Y_{t_2} = -1.19$, $Y_{t_5} = -5.61$, $Y_{t_6} = -4.03$, $Y_{t_7} = 3.81$, $Y_{b_1} = 0.024$, $Y_{b_2} = 0.5$, $Y_{b_3} = 1.75$, $Y_{b_4} = 0.64$, $Y_{b_5} = 0.02$, $m_1 = -4.8$ TeV , $m_2 = -3.12$ TeV, $m_3 = 1.11$ TeV, $m_4 = 1.5$ TeV and $m_5 = 1.1$ TeV, with $Y_{b_2} = 0.52$ and $Y_{b_3} = 1.75$ in the top sector plot (left) plus $Y_{t_3} = -4.59$ and $Y_{t_4} = -4.51$ in the bottom sector plot (right).

We observe from the three sets of plots Figs. 1, 2, and 3 that the allowed regions in the $[Y_{t_3},Y_{t_4}]$ and the $[Y_{b_2},Y_{b_3,b_4}]$ planes are not small. Nevertheless, other choices of the remaining parameters do not allow one to increase largely those domains. In general, besides $S$ and $T$, the most important constraint in the top sector come from enforcing the enhancement of the $t \bar{t} H$ rate without significantly altering the top-quark mass. Simultaneously respecting these two constraints calls for a strong mixing with the extra quarks, which translates into larger Yukawa couplings $Y_{t_i}$, with $i=1,2,3,\ldots$. As a consequence, the allowed regions are driven close to the areas ruled out by non-perturbativity, with the highest Higgs–VLQ couplings reaching values typically higher than 3 (for model C, the allowed region only touches the area ruled out by non-perturbativity). Another possibility of enhancing the mixing would be to lower the VLQ mass parameters $m_i$, but this approach fails, since it leads to VLQ masses that are too low and experimentally excluded. Concerning the bottom sector, the strongest constraints come clearly from the LEP observables $A_\mathrm{FB}^b$ and $R_b$, which are measured at the per mille level, as well as from the $S$ and $T$ oblique parameters, measured (indirectly) also at LEP.

Fig. 4

Regions in the $[ c_{t}, m_{t_2} ]$ plane (upper plots) and the $[ c_{b}, m_{b_2} ]$ plane (lower plots) where all the phenomenological constraints enlisted in the previous section are satisfied. The varied parameters and their corresponding ranges for the three models are given in the text

Note that our choice for presenting the results (Figs. 1, 2, 3) in the $[Y_{t_3},Y_{t_4}]$ and the $[Y_{b_2},Y_{b_3,b_4}]$ planes was only for illustration, as any other choice would have been equally valid. Nonetheless, when searching for the allowed regions, we varied all the parameters involved in our models. Even though the parameter space of each of our models has a dimensionality larger than 10, varying by a significant amount the couplings $Y_{t_i,b_i}$ and/or the VLQ mass parameters $m_i$, which we kept fixed in the plots would have restrained the starred regions, where all the constraints are satisfied. Thus, it was not possible to decrease the large values of some of the interaction basis couplings, which were $Y_{t_i,b_i}>3$. Nevertheless, we remind the reader that the displayed regions involve only perturbative couplings in the mass basis.

Interestingly, the considered models predict the existence of top, bottom ($t_2$ and $b_2$ eigenstates) and even exotic partners around the TeV scale, to which the LHC Run II might be sensitive. While model B predicts 7 VLQs with masses $\lesssim$2 TeV, models A and C both predict 4 VLQs with masses $\lesssim 2$ TeV. Such states will be thus accessible through direct production at the upgraded LHC.

Another feature that can be noticed from the plots is the fact that in the allowed regions, the top-quark mass attains rather large values, usually above $185$ GeV, while the $t \bar{t} H$ signal strength has a value around $1.65$, which is approximately $1\sigma$ below its central value, 2.23. In fact, the considered VLQ models can more closely reproduce simultaneously the measured top mass $m_t \simeq 174$ GeV and a higher value of the $t \bar{t} H$ signal strength, typically $\mu _{ttH} \simeq 2$ (i.e. only $\sim$ 0.3$\sigma$ away from the central value), but at the expense of having $S$ and $T$ values outside their $2 \sigma$ ranges. One can argue that $S$ and $T$, which are measured with a higher accuracy than the Higgs couplings, could also be sensitive to the presence of other sources of new physics, such as extra gauge bosons that appear in many scenarios with VLQs,⁶ allowing to increase the range of validity of the Yukawa couplings with the data in Figs. 1, 2 and 3.

Note that the three VLQ models that we consider improve the discrepancies in $A_\mathrm{FB}^b$ not only on the Z-pole but also off the Z-pole. For instance, in model A, for the allowed region of the parameter space in the lower part of Fig. 1, the $\chi ^2$ function of the fit of all the asymmetry measurements is reduced from $\chi ^2_\mathrm{SM} \simeq 33$ down to typically $\chi ^2_\mathrm{VLQ} \simeq 15$.

To summarize the discussion of this section, we present in Fig. 4 a “summary plot” containing, for each considered model, the predicted values of $c_{t} \equiv |y_{t_1} / y_t^\mathrm{SM}|$ and $m_{t_2}$ (upper plots) plus $c_{b} \equiv |y_{b_1} / y_b^\mathrm{SM}|$ and $m_{b_2}$ (lower plots), where $y_{Q_1}$ is the Yukawa coupling (in the mass basis) of the $Q_1$ mass eigenstate, i.e. the observed top and bottom quarks, and $y_Q^\mathrm{SM}$ is the SM prediction (the two values are equivalent in the interaction or mass basis if no fermion mixing is present). Thus, $c_t$ and $c_{b}$ measure the relative departure from the SM of the Yukawa couplings of the top and bottom quarks. As already mentioned throughout the paper, $m_{t_2}$ ($m_{b_2}$) represents the mass of the lightest top-like (bottom-like) VLQ in each of the three obtained models. For completeness, we also quote, for the allowed regions in each model, the typical masses of the lightest electric charge $Q= - \frac{4}{3}, \frac{5}{3}, \frac{8}{3}$ VLQs:

• In model A, $m_{5/3_1} = |m_1| \simeq 1.77$ TeV and $m_{4/3_1} = |m_3| \simeq 0.85$ TeV;

• In model B, $m_{5/3_1} = |m_1| \simeq 1.42$ TeV and $m_{4/3_1} \simeq 1.07 - 1.1$ TeV;

• In model C, $m_{8/3_1} = |m_2| \simeq 3.12$ TeV, $m_{5/3_1} \simeq 2.95 - 3$ TeV, $m_{4/3_1} = |m_4| \simeq 1.5$ TeV. Note that, in this model, the bounds mentioned in Sect. 3.3, $m_{8/3} > 940$ GeV and $m_{5/3} > 1$ TeV, are both respected.

In the figure, the varied parameters and their corresponding variation ranges are $Y_{t_3} \in [-1.6,-1.2]$, $Y_{t_4} \in [4,4.6]$, and $Y_{b_2} \in [2,4]$, $Y_{b_3} \in [0.7,1.2]$ for model A, $Y_{t_3} \in [-3.4,-3]$, $Y_{t_4} \in [-1.4,-1]$, and $Y_{b_2} \in [1,2]$, $Y_{b_4} \in [0,1]$ for model B, plus $Y_{t_3} \in [-5,-4.2]$, $Y_{t_4} \in [-5.5,-3.5]$, and $Y_{b_2} \in [-0.8,0.8]$, $Y_{b_3} \in [1.3,2.2]$ for model C. These intervals cover roughly the allowed regions in Figs. 1, 2 and 3 and, for each model, the remaining parameters are fixed at the same values as in these figures. Obviously, the Yukawa couplings with “t” and “b” subscripts correspond, respectively, to the top and bottom sectors.

The two quantities $c_t$ and $c_{b}$ are defined as absolute values as the sign of the Yukawa couplings in the mass basis is not physical. Instead, the signs of $y_{t_1}/m_{t_1}$ and $y_{b_1}/m_{b_1}$ are of physical relevance since such ratios appear directly in the loop-mediated $gg \rightarrow H$ and $H \rightarrow \gamma \gamma$ amplitudes (see the appendix for a discussion). For example, in the $H \rightarrow \gamma \gamma$ process, a negative $y_{t_1}/m_{t_1}$ ratio would mean that the top-quark loop amplitude would interfere constructively with the W-loop amplitude, leading to an increase of $\Gamma ( H \rightarrow \gamma \gamma )$ with respect to the SM value. In principle, this is possible in general VLQ scenarios but it is not the case in our chosen models. In the regions where all phenomenological constraints are satisfied, we find that in the three models $y_{t_1}/m_{t_1}$ is positive, as in the SM, but slightly higher as a result of the enhancement of the top Yukawa coupling. Depending on the model, the new VLQ mass eigenstates propagating in the loop interfere either constructively or destructively with the top-quark exchange. Nevertheless, their contribution to the triangular loop is modest since their masses are rather large and their couplings to the Higgs boson are small, being induced only through quark mixing.⁷

As a final remark, there is no complete cancellation between the effects of the enhanced top Yukawa coupling and the contribution of the new VLQ states in the triangular loop. Instead, it turns out that in each of the models that we have considered, the gluon fusion cross section is increased by 10–15 % compared to the SM value. Meanwhile, relative to its SM value, the diphoton partial width is suppressed by 1–2 % in models A and B, whereas in model C it is enhanced by $\sim$15 %. At present, these slight deviations from the SM are below the experimental accuracy on the signal strengths measured by the ATLAS and CMS collaborations [61–63].

5 The sensitivity on VLQs at the upgraded LHC

We now turn to the discussion of the sensitivity on VLQs that could be achieved at the upgraded LHC with $\sqrt{s}=14$ TeV c.m. energy when 3000 fb$^{-1}$ of data will be collected, the so-called high-luminosity option of the LHC (HL-LHC) and start with a discussion of the observables that can be measured with high precision in this case.

5.1 Precision Higgs observables at high-luminosity

Compared to $\sqrt{s}=8$ TeV, the Higgs production cross sections at $\sqrt{s}=14$ TeV are enhanced by a factor of approximately 2.5 in the case of gluon fusion, 2 in the case of Higgs-strahlung and 5 in the case of associated $t \bar{t} H$ production. The statistical uncertainties on the measurement of the signal strengths values $\mu _{XX}$ for the various processes listed in Sect. 3.1 and obtained at $\sqrt{s}=7+8$ TeV with $\sim$25 fb$^{-1}$ data, will be thus significantly reduced at this LHC upgrade. For instance, in the ggF mode, the statistical error which is presently the largest uncertainty will be reduced by a factor $\sqrt{300} \approx 15$ with 3000 fb$^{-1}$ data and would lead to a precision of the order of 1–2 % in the case of the $\mu _{\gamma \gamma }$ and $\mu _{ZZ}$ signal strengths and 3–5 % in the case of $\mu _{bb}$. The smaller systematical uncertainties could also be reduced so that one could hope that the total experimental errors would be reduced to the few percent level in accord with the ATLAS and CMS projection at $\sqrt{s}=14$ TeV with 3000 fb$^{-1}$ data [76, 77].

The theoretical uncertainties that affect the production cross sections (which are at the level of 10 % in the ggF and 5 % in the VH cases for instance) and the decay branching ratios (which are presently of order 5 % in most channels) would turn then to be the largest source of uncertainty and would limit the interest of these measurements if they are not significantly reduced. Nevertheless, one could construct ratios of observables that are free of these uncertainties. In particular, the ratio of production times decay rates [70, 71]

$$\begin{aligned} D_{\gamma \gamma }&= \frac{\sigma (pp\rightarrow H \rightarrow \gamma \gamma )}{\sigma (pp\rightarrow H \rightarrow ZZ^*)} \simeq \frac{\Gamma (H \rightarrow \gamma \gamma )}{\Gamma (H \rightarrow ZZ^*)}, \end{aligned}$$

(23)

$$\begin{aligned} D_{bb}&= \frac{\sigma (q\bar{q} \rightarrow VH \rightarrow V b\bar{b})}{\sigma (q\bar{q} \rightarrow VH \rightarrow V WW^*)} \simeq \frac{\Gamma (H \rightarrow b\bar{b})}{\Gamma (H \rightarrow WW^*)}, \end{aligned}$$

(24)

will be free of all these theoretical uncertainties (including also possible ambiguities in the Higgs total decay width that affect all the branching ratios) provided that the fiducial cross sections for the processes in the numerator and in the denominator are measured within the same kinematical configurations. The two observables will then be limited only by the experimental error and, in particular, the statistical one (at least for $D_{\gamma \gamma }$). At the HL-LHC, one expects that accuracies of the order of

$$\begin{aligned} \Delta D_{\gamma \gamma } \approx 1\,\%\quad \mathrm{and}\; \Delta D_{bb} \approx 5\,\% \end{aligned}$$

(25)

could be achieved. The decay ratios above, which measure only the ratio of Higgs couplings squared $g_{HXX}^2$, would then be extremely powerful tools to indirectly probe new physics effects and, in particular, those of heavy VLQs of the third generation.

Another Higgs decay ratio which could also be very useful in general is $D_{\tau \tau }= \Gamma (H \rightarrow \tau \tau )/ \Gamma ( H \rightarrow WW^*$), with the Higgs state produced in the ggF+1j and VBF modes. However, we will ignore it in our discussion, since the VLQs that we are analyzing here do not affect the $H\tau \tau$ and HVV couplings and will thus have no impact in this context.

Finally, the signal strength in the associated Higgs production with top-quark pairs, $pp\rightarrow t\bar{t}H$, is also important in the context of VLQs. At the HL-LHC, both the ATLAS and the CMS collaborations expect a measurement of the cross section $\sigma (pp \rightarrow t\bar{t} H)$ with an experimental accuracy of the order of 15 % [76, 77]. This error is largely dominated by the statistical one. In addition to that, the process, which is known at NLO in the QCD and electroweak couplings [93–96], is affected by a theoretical uncertainty of about 15–20 % from the variation of the renormalization and factorization scales and from the parton distribution functions and the value of $\alpha _s$. This leads then to a total uncertainty of about 30 %. Nevertheless, it has been advocated that considering the ratio of cross sections for associated $t \bar{t} H$ and $t \bar{t} Z$ boson production,⁸ $C_{tt}= \sigma (pp\rightarrow t\bar{t} H)/\sigma (pp \rightarrow t\bar{t} Z)$, will also significantly reduce the theoretical uncertainties to the level of $\sim$5 % [97]. One would then have a total error on the ratio at the level of 15 % when combining the ATLAS and CMS measurements at HL-LHC.

Hence, the ratio $C_{tt}$ is expected to be affected by a much larger error than the $D_{\gamma \gamma }$ and even $D_{bb}$ ratios, thus reducing its capacity to probe tiny VLQ effects. For this reason, although providing a complementary information as it is exclusively sensitive to the t–$t'$ mixing, we will not include this ratio in the rest of our VLQ analysis.

5.2 Probing VLQs using the Higgs decay ratios

Using the $D_{\gamma \gamma }$ and $D_{bb}$ decay ratios, with the total uncertainties given in Eq. (25) and their projected central values equal to their SM values, we now estimate the sensitivities that could be achieved on VLQs at the HL-LHC. In this analysis, it would be useful to simplify to a certain extent the previously considered models in order to keep the discussion as transparent as possible but still at a rather general level. We will thus make the following three simplifying assumptions.

First, since we would like to study the new physics effects only and not the mixing effects between the SM and the physics beyond it, we will assume the VLQs to decouple from the top and bottom quarks, thus leaving the latter’s couplings to the Higgs boson SM-like. This is a good approximation in general since the VLQs that we are investigating have masses well above the electroweak scale and, thus, are supposed to mix weakly with the SM states. At this stage, we will no longer attempt to explain the LHC hint for an increased top Yukawa coupling nor the anomaly in the $A_\mathrm{FB}^b$ asymmetry. We will thus allow the new physics that we are considering to communicate with the SM only via the Higgs boson, an assumption which guarantees that the models which we are investigating comply with the currently available phenomenological constraints.⁹ For all the other phenomenological constraints, in particular for the electroweak oblique observables S and T, we assume the same central values and errors as presently (we thus ignore for simplicity some potential improvement such as the one that would come from a better measurement of the W boson mass at the LHC). The “theoretical” constraints from the top- and bottom-quark masses and from the perturbativity of the Yukawa couplings, as well as the lower bounds on the masses of the VLQs (which might be improved by the time of the HL-LHC if no signal is found, but will be superseded by the limits that will be obtained in our analysis) will also be assumed to be the same.

Second, to focus as much as possible on the effect of a single VLQ and not consider the cumulative contribution of several ones (for instance in the contributions to the $H \rightarrow \gamma \gamma$ or $gg \rightarrow H$ loop processes), we will retain for each model only two vector-like multiplets and decouple completely the others. The reason to retain two multiplets and not only one is that at least two fields are needed to have interactions with the Higgs boson. This interaction with the Higgs field generates, after electroweak symmetry breaking, a mixing term between the two vector-like fields. However, to still concentrate on the effect of a single VLQ, we consider the VL mass parameter of one of the two multiplets to be larger than the other (this guarantees a small effect of the heavier VLQ in the loop-induced $H\gamma \gamma$ and Hgg vertices for instance). Nevertheless, at the same time, this mass splitting significantly reduces the Yukawa coupling of the lighter VLQ as a result of mixing factors. In this case, only a one percent measurement of the $D_{\gamma \gamma }$ ratio could signal the new physics effects. Our goal will be simply to estimate the power of high-precision measurements in the Higgs sector to probe heavy VLQ states with small couplings to the Higgs bosons.

Finally, also for simplicity reasons, we will assume that the two possible Higgs–VLQ–VLQ couplings in the interaction basis are equal, which means that, in the same basis, the VLQ mass matrices are symmetric. The latter have the simple texture

$$\begin{aligned} \mathcal {M}_\mathrm{VLQ} = \left( \begin{array}{cc} m &{} m_Y \\ m_Y &{} M \end{array}\right) . \end{aligned}$$

(26)

In this expression, $m\;(M$) is the lighter (heavier) VLQ mass parameter, while $m_Y$ is, up to a possible Clebsch–Gordan factor, equal to $v' Y$ (as the highest multiplet we consider is a triplet, $1/\sqrt{2}$ is the only possibility for a Clebsch–Gordan coefficient). In each model, M will be fixed to a high value, while m and Y will be treated as variable parameters.

In the three discussed models, the various multiplets that we retain and their impact on the $D_{\gamma \gamma }$ ratio and hence on the $H\gamma \gamma$ loop are as follows (as already stated, VLQ states are also exchanged in the loop-induced ggF production mechanism but the production rates cancel in the $D_{\gamma \gamma }$ ratio):

• In model A, $(q_{5/3}, t')$ is the lighter doublet, with mass parameter m, while the heavier VL field (with the larger mass parameter M) is the $t''$ singlet. Both top-quark partners will enter in the $H\gamma \gamma$ loop and affect the amplitude.

• In model B, the $(b', q_{4/3})$ doublet is the lighter multiplet while the heavier one is the $(t'',b'', q'_{4/3})$ triplet. Here, the main players will be the exotic $q_{4/3}$ states, while the bottom-quark partners would generate a tiny effect on the triangular Higgs-diphoton loop, of order $(Q_\mathrm{em}(q_{4/3}) / Q_\mathrm{em}(b))^2 = 16$ times smaller than the contribution of the electrically charged $-4/3$ quarks.

• For model C, the $(q_{5/3}, t')$ doublet has a mass parameter m and the $(q_{8/3},q'_{5/3}, t'')$ triplet, a mass M. Here, the main contribution will be that of the exotic $q_{5/3}$ states. The contribution of the top-quark partners is approximately $\frac{1}{\sqrt{2}} (Q_{5/3} / Q_{top})^2 \simeq 4.42$ times smaller than that of the $q_{5/3}$ states ($1/\sqrt{2}$ is a Clebsch–Gordan).

In each of these cases, the eigenmass of the lighter VLQ will be denoted by $m_\mathrm{VLQ}$, and its coupling (in the same mass basis) to the Higgs boson by $y_\mathrm{VLQ}$. These two quantities are deduced from the diagonalization of the matrix in Eq. (26). Due to the fact that $M \gg m$, one has $m_\mathrm{VLQ} \sim m$ and $y_\mathrm{VLQ} \sim -v Y^2 / (M-m)$. Some technical aspects concerning our $D_{\gamma \gamma }$ analysis can be found in the appendix.

For the case of the $Hb\bar{b}$ vertex, which can be probed directly in the measurement of the $H\rightarrow b\bar{b}$ partial width, the discussion concerning the $D_{bb}$ decay ratio will be even simpler. Here, we will consider a non-vanishing mixing between the b quark and its VL partners. In turn, we will consider only bottom-like VLQs since only such states affect the $Hb\bar{b}$ coupling through b–$b'$ mixing. For simplicity, we shall consider the case of only one bottom-like VL partner $b'$. The choice of the $\mathrm SU(2)_L$ representation of the $b'$ extra quark will be qualitatively irrelevant due to similar mass matrix textures (the only quantitative difference could come from various Clebsch–Gordan factors, depending on the $\mathrm SU(2)_L$ embedding of $b'$). Thus, a single picture could be representative of all three considered models. By minimality, we shall take the $b'$ as a singlet under $\mathrm SU(2)_L$, which, together with the SM b quark, will lead to a mass matrix given by

$$\begin{aligned} \mathcal {M}_{b} = \left( \begin{array}{cc} m_{Y_1} &{} m_{Y_2} \\ 0 &{} M \end{array}\right) , \end{aligned}$$

(27)

with $m_{Y_{1,2}} \equiv v' Y_{1,2}$. We shall denote by $y_{b'}$ the Higgs-VLQ coupling in the mass basis and by $m_{b'}$ the bottom-like VLQ eigenmass, both being obtained from the bi-diagonalization of the mass matrix in Eq. (27).

In the analysis, we will treat $Y_2$ and M, defined in Eq. (27), as variable parameters. The remaining parameter, $Y_1$, also appearing in Eq. (27), will be expressed in terms of $Y_2$ and M by demanding that $m_b$, the observed b quark mass, is reproduced. Since $M \gg m_{Y_{1,2}}$ in most of the interesting part of the parameter space, we have, to a very good approximation, $m_b \approx m_{Y_1} (1 - {m_{Y_2}^2}/{2M^2})$, which can easily be inverted in order to re-express $Y_1$ as a function of $Y_2$ and M. For this purpose, the value of the bottom-quark mass in our numerical analysis will be taken to lie between the $\overline{\mathrm{MS}}$ value $m_b ({\overline{\mathrm MS}}) \approx 4.18$ GeV, and the on-shell value $m_b(1 \mathrm {S}) \approx 4.65$ GeV [57], with a mean value $m_b = 4.43$ GeV. Apart from this constraint, we shall enforce the perturbativity condition of the mass basis Yukawa couplings, $y \lesssim \sqrt{4\pi }$, and the LHC bottom-like VLQ exclusion limit, $m_{b'} > 813$ GeV [50].

Fig. 5

Regions in the $[ m_\mathrm{VLQ},y_\mathrm{VLQ} ]$ plane for the simplified versions of models A, B and C, to which a precise measurement of $D_{\gamma \gamma }$ at the HL-LHC, with $\Delta D_{\gamma \gamma } = 1\,\%$, will be sensitive. The other parameters entering the analysis are discussed in the text

We display in Fig. 5, for the simplified versions of models A-C, regions in the plane $[m_\mathrm{VLQ},y_\mathrm{VLQ}]$ to which a precise measurement with $\Delta D_{\gamma \gamma } = 1\,\%$ will be sensitive. In this figure, we have assumed that the future central experimental value of $D_{\gamma \gamma }$ would be equal to its SM prediction. The choices for the heavy VLQ mass parameters are $M_A = 15$ TeV, $M_B = 25$ TeV and $M_C = 28$ TeV. For each model, the ranges of the parameters are $m \in [-15, 15]$ TeV and $Y \in [0, 5]$. The lower boundary of each region is given by the $Y = 5$ curve, which typically marks the transition to the non-perturbativity regime, while the upper boundary is dictated by the $\Delta D_{\gamma \gamma } = 1\,\%$ condition. The region defined by $\left| m_\mathrm{VLQ} \right| \lesssim 0.8$ TeV, delimiting the third boundary, is excluded by direct VLQ searches.

Similarly, we present in Fig. 6 regions of the $[m_{b'},y_{b'}]$ plane to which a 5 % accuracy measurement of $D_{bb}$ will be sensitive. In this figure, we have assumed, as in the case of $D_{\gamma \gamma }$, that the future central experimental value of $D_{bb}$ would be equal to its SM prediction. Here, the ranges of the parameters are $M \in [0.5, 6]$ TeV and $Y_{b_2} \in [0,6]$. The lower boundary of the obtained region is determined by the $\Delta D_{bb}= 5\,\%$ condition, while the upper right one signals the passage to non-perturbativity. The upper left boundary delimits the zone where the observed bottom quark becomes too light, whereas the left boundary shows the lower limit $m_{b'} \lesssim 0.8$ TeV from direct searches of b-like VL partners.

Fig. 6

Regions in the $[ m_{b'},y_{b'} ]$ plane for a simplified VLQ model to which a precise measurement of $D_{bb}$ at the HL-LHC, with $\Delta D_{bb} = 5\,\%$, will be sensitive. The other parameters entering the analysis are discussed in the text

Figures 5 and 6 constitute our main prospective results and one can see that VLQ masses up to several TeV can be probed. With the precise measurement of $D_{\gamma \gamma }$ top partners with masses up to 5  TeV can be resolved in the loop, while exotic quarks (with $Q_\mathrm{em} = - \frac{4}{3}, \frac{5}{3}$) with masses as high as $\sim$13 TeV are probed. Meanwhile, a 5 % error in the measurement of $D_{bb}$ can be sensitive to the presence of bottom-like VLQs with masses up to $\sim$5 TeV. It is interesting to observe the complementarity between the two measurements: while with $D_{\gamma \gamma }$ one can efficiently resolve multi-TeV scale top and exotic VL partners, very heavy bottom VL partners can be probed through $D_{bb}$.

As expected, $D_{\gamma \gamma }$ is more sensitive to the VLQs with higher electric charge that occur in models B and C. The mass limits above are much higher than the ones obtained from direct VLQ searches which, even at the HL-LHC, would only reach the 2 TeV range [76]. It may be surprising that the mass reach for the $q_{4/3}$’s of model B is higher than the one for the $q_{5/3}$’s of model C, but the explanation is simple. As it is visible from the figure, model C has a lower sensitivity on $m_\mathrm{VLQ}$ but for a lower coupling $y_\mathrm{VLQ}$. The relative smallness of the couplings in model C has two reasons: on the one hand, the Yukawa couplings for model C are suppressed by a Clebsch–Gordan factor of $1 / \sqrt{2}$ and, on the other hand, the mass parameter M is larger in model C ($M \sim 28$ TeV) than in model B ($M \approx 25$ TeV), which leads to a smaller mixing between the two VLQs and hence a smaller coupling for the lighter ones to the Higgs boson.

We should also mention that, in the $D_{\gamma \gamma }$ discussion, for models B and C, the oblique parameters S and T are well within $2\sigma$ for all values of m and Y not excluded by non-perturbativity or by direct searches of VLQs. The situation is not as good in model A, where, for $m_\mathrm{VLQ} \le 3$ TeV, S and T deviate by more than $3\sigma$. However, since we are interested in knowing the highest possible VL mass that can be resolved in the $H \rightarrow \gamma \gamma$ loop, this is not a serious problem. The case of $D_{bb}$ is similar to the one in model A: for $m_{b'} \gtrsim 3$ TeV, S and T are within $2\sigma$ from their central values.

6 Conclusions

We have analyzed in this paper the sensitivity of present and future LHC Higgs data to heavy vector-like partners of the top and bottom quarks that appear in many extensions of the SM, such as warped extra-dimension scenarios and composite Higgs models. Working in an effective approach and considering several VLQ representations under the SM gauge group, we have thoroughly investigated three models that address simultaneously the long-standing puzzle of the forward–backward asymmetry $A_\mathrm{FB}^b$ at LEP and the recently observed deviation from its SM value of the cross section of the $pp \rightarrow t\bar{t}H$ production process at the LHC. On the other hand, the three models fulfill all other experimental and theoretical constraints, in particular those coming from the electroweak precision measurements and from the LHC data in the Higgs decay and main production channels.

We have used the principle of minimality as a guide to select representative examples of the $t'$ and $b'$ multiplets, which should be related through their contributions to the highly constrained electroweak precision data and address the two aforementioned anomalies. Among the multiplets that involve $t',b'$ and VLQs with exotic electric charge, one has, for example, $t'$, $b'$ singlets, $(q_{5/3},t')$, $(b', q_{4/3})$ doublets and/or $(t',b',q_{4/3})$, $(q_{8/3}, q_{5/3},t')$ triplets. These states mix with the SM top and bottom quarks and modify their Yukawa and gauge couplings. In addition, they would contribute to the loop-induced $gg\rightarrow H$ production and $H\rightarrow \gamma \gamma$ decay processes. For instance, the mixing with the additional states in the bottom sector allows for a sufficiently large increase of the $Z b_R b_R$ coupling to explain the $A_\mathrm{FB}^b$ anomaly. At the same time, an enhancement of the $Ht\bar{t}$ Yukawa coupling by a factor up to $\sim$1.4 can occur, which would instead explain the $\sim$2$\sigma$ apparent increase of the cross section $\sigma (pp\rightarrow t\bar{t} H)$ at the LHC. The rates for the loop-induced processes would stay SM-like due to either small VLQ contributions or compensating effects between fermion mixing and loop contributions. Interestingly, the considered models predict the existence of VLQ with masses in the range 1–2 TeV, which might be discovered at the current Run II of the LHC with a c.m. energy of 13–14 TeV.

In a second part of the paper, we left aside the anomalies in the asymmetry $A_\mathrm{FB}^b$ and the cross section $\sigma (pp \rightarrow t\bar{t}H)$ and focused instead on the VLQ mass scale that could be probed in the future by precision measurements in the Higgs sector at the high-luminosity LHC option. In this context, the ratios of the partial widths of the $H\rightarrow \gamma \gamma$ vs. $H\rightarrow ZZ^*$ and $H\rightarrow b\bar{b}$ vs. $H\rightarrow WW^*$ decay modes, $D_{\gamma \gamma }$ and $D_{bb}$, would play an important role as they can be determined with an accuracy at the level of, respectively, $\Delta D_{\gamma \gamma } = 1\,\%$ and $\Delta D_{bb} = 5\,\%$. Assuming the worst-case scenario in which the new physics scale would lie far above the electroweak scale and all other measured observables would appear to be SM-like, we have shown that, in some simplified VLQ frameworks, the precise measurement of the two decay ratios would probe VLQs with masses above the multi-TeV range. In particular, VLQs contributing to the $H\gamma \gamma$ loop vertex or altering at tree level the $Hb\bar{b}$ coupling would be visible at the HL-LHC if the mass scales are $\sim$5 TeV for top and bottom partners and up to $\sim$13 TeV for VLQs with higher electric charge, such as $-\frac{4}{3}$ or $\frac{5}{3}$. These mass values are much higher than those attainable in direct VLQ searches at the LHC in the present [49–51] or even in the future [76, 77].

Appendix: VLQ contributions to the Hgg and \(H\gamma \gamma \) vertices

Noting the Yukawa couplings for the VL mass eigenstates as $y_{t_i}$ and $y_{b_i}$ (labeled by $i=1,2,\ldots$), the ratio of the ggF cross section over its SM prediction reads

$$\begin{aligned} \mu _\mathrm{ggF} \equiv \frac{\sigma ^\mathrm{VL}_\mathrm{ggF}}{\sigma _\mathrm{ggF}^\mathrm{SM}} \ = \ \frac{\bigg \vert \sum _i \ \frac{v \, y_{t_i}}{m_{t_i}} A[\tau (m_{t_i})] + \sum _i \ \frac{v \, y_{b_i}}{m_{b_i}} A[\tau (m_{b_i})] \bigg \vert ^2}{\bigg \vert A[\tau (m_{t})] + A[\tau (m_{b})]\bigg \vert ^2} \, \end{aligned}$$

(A.1)

where $A[\tau (m)]$ is the form factor for spin 1/2 particles [81] normalized such that $A[\tau (m)\ll 1]\rightarrow \frac{4}{3}$ and $A[\tau (m)\gg 1]\rightarrow 0$ with $\tau (m)=m_H^2/4m^2$. It is useful to use this large mass limit, which is a reasonable approximation (except for the bottom quark), so that the first sum from Eq. (A.1) simplifies [98],

$$\begin{aligned} \sum _i \ \frac{v y_{t_i}}{m_{t_i}} A[\tau (m_{t_i})]\simeq & {} \sum _i \frac{v y_{t_i}}{m_{t_i}} = v \, \mathrm{Tr}\left( \frac{\partial \mathcal{M}_t}{\partial v} \mathcal{M}^{-1}_t\right) \nonumber \\= & {} v \frac{\partial }{\partial v} \log \det \mathcal{M}_t . \end{aligned}$$

(A.2)

Eq. (A.2) is useful as, due to the invariance of the trace with respect to basis changing, it can be applied to the matrix $\mathcal {M}_t$ in the starting interaction basis as well, thus avoiding the explicit calculation of the basis transformation. A similar trick can be used for the sum over the bottom-quark states. However, since the bottom form factor is almost zero, one has to add and then subtract its contribution:

$$\begin{aligned} \sum _i \ \frac{v y_{b_i}}{m_{b_i}} A[\tau (m_{b_i})]\simeq & {} \sum _i \frac{v y_{b_i}}{m_{b_i}} - \frac{v y_{b_1}}{m_{b_1}}\nonumber \\= & {} v \frac{\partial }{\partial v} \log \det \mathcal{M}_b - \frac{v y_{b_1}}{m_{b_1}}, \end{aligned}$$

(A.3)

which enables us to write an approximate form for the ggF ratio from Eq. (A.1):

$$\begin{aligned} \mu _\mathrm{ggF} \simeq v^2 \left| \frac{\partial }{\partial v} \log \det \mathcal{M}_t + \frac{\partial }{\partial v} \log \det \mathcal{M}_b - \frac{y_{b_1}}{m_{b_1}} \right| ^2. \end{aligned}$$

(A.4)

Additionally, in order to use the above equation as a transparent guide to start the exploration of a multivariate parameter space, one can take the $v y_{b_1} / m_{b_1}$ term to be $\mathcal {O}(1)$, just as in the SM. Moreover, if there are other higher charged exotic quarks that couple to the Higgs boson, a term of the type $\frac{\partial }{\partial v} \log \det \mathcal{M}_X$, with $X$ denoting the quark type, should be added to the previous equation. Thus, Eq. (A.4) can be used as a guide to stay in regions where the ggF rate is not too far from its SM value. Nevertheless, in our numerical analyses, we use the exact expression of Eq. (A.1).

As for the $R_{\gamma \gamma }$ ratio (which is different from $D_{\gamma \gamma })$,

$$\begin{aligned} R_{\gamma \gamma } \equiv \frac{\Gamma (H \rightarrow \gamma \gamma )}{\Gamma (H \rightarrow \gamma \gamma )_{SM}}, \end{aligned}$$

(A.5)

which enters in $\mu _{\gamma \gamma }$, a similar formula can be derived [81], and the tricks displayed above can be used once again. The difference is that the W-boson also runs in the loop, which generates an additional term besides the ones in Eq. (A.4), and that each sum over quarks gets multiplied by $N_C = 3$ and by $(Q_{e.m.}(q))^2$, where $Q_{e.m.}(q)$ is the $\mathrm {U(1)_{em}}$ charge of the quark in question. Therefore, the bottom-quark contribution becomes negligible, hence allowing one to reliably estimate $R_{\gamma \gamma }$ by differentiating with respect to $v$ the various $\log \det \mathcal {M}_q$ terms.

We turn now to some technical aspects concerning the study of $D_{\gamma \gamma }$ in the context of our simplified VLQ models. The values for the interaction basis parameters (Y, m, and M) and for the mass basis quantities ($m_\mathrm{VLQ}$ and $y_\mathrm{VLQ}$) were chosen on the following grounds. First of all, Y was chosen to be always positive because its sign has no impact on the physics. To understand more easily the nature of Y’s sign, let us denote the lighter VLQ as q and the heavier one as Q. There exists a $\mathbb {Z}_2$ field transformation under which, for example, q is odd and Q is even. Performing such a transformation on the two VLQ fields, which leaves the VLQ kinetic and mass terms invariant, would change the sign of the $Y (\bar{Q} q + \mathrm {H.c.})$ terms of the Lagrangian. By virtue of this transformation, Y can always be set to a positive value, which means that its sign is not physically meaningful. Also, we have chosen $Y \le 5$ because, for $Y \gtrsim 5$, at least one of the four Yukawa couplings in the mass basis becomes non-perturbative, i.e. greater than $\sim \sqrt{4\pi }$. Moreover, while m was allowed to attain both negative and positive values, M was chosen to be positive for all the models, since only the relative sign of the two parameters influences the physics.

The explanation reads as follows. Since we assume that there is no mixing between the SM fields and the VLQs, we have for the amplitude of the $H \rightarrow \gamma \gamma$ process

$$\begin{aligned} \mathcal {A}_{\gamma \gamma } = \mathcal {A}_\mathrm{SM} + \mathcal {A}_\mathrm{VLQ}, \end{aligned}$$

(A.6)

where $\mathcal {A}_\mathrm{SM}$ and $\mathcal {A}_\mathrm{VLQ}$ are, respectively, the SM state-mediated and VLQ-mediated loop amplitudes. As explained in the beginning of this appendix, in the limit where the VLQs are much heavier than the Higgs boson,

$$\begin{aligned} \mathcal {A}_\mathrm{VLQ} \propto \frac{\partial }{\partial v} \left[ \log \left( \det \mathcal {M}_\mathrm{VLQ} \right) \right] . \end{aligned}$$

(A.7)

Thus, $A_{\gamma \gamma }$ depends only on $\frac{\partial }{\partial v} [ \log ( \det \mathcal {M}_\mathrm{VLQ} ) ]$ and, therefore, the sign of a parameter is physically relevant only if it affects the VLQ mass matrix determinant, which is given by

$$\begin{aligned} \det \mathcal {M}_\mathrm{VLQ} = mM - m_Y^2. \end{aligned}$$

(A.8)

This expression shows that only the sign of mM enters in the studied observable quantity and confirms that the sign of Y is unphysical (recall that $m_Y \propto Y$). Finally, the values of M for each model were taken such that the largest resolvable $m_\mathrm{VLQ}$ is roughly half of M, which avoids too much feedback in the $H \rightarrow \gamma \gamma$ loop from the heavier VLQ and thus isolates to some extent the contribution to $D_{\gamma \gamma }$ of the lighter VLQ.

As for the mass basis quantities, $y_\mathrm{VLQ}$ and $m_\mathrm{VLQ}$, the discussion is somewhat simpler. Only their relative sign is of physical importance, since they enter $D_{\gamma \gamma }$ through their ratio, as it appears for example in Eq. (A.1) for the comparable structure of the ggF loop amplitude. Therefore, we plotted only negative values for $y_\mathrm{VLQ}$, while letting $m_\mathrm{VLQ}$ have any sign. Nevertheless, we did not plot the region where $| m_\mathrm{VLQ} | \le 0.8$ TeV (the empty band along the $m_\mathrm{VLQ}=0$ line), since it is excluded by direct searches for VL partners.

Also, it is interesting to note in Fig. 4 the asymmetry of the regions with respect to the $m_\mathrm{VLQ} = 0$ axis. This is due to the fact that, in the case of $m_\mathrm{VLQ} > 0$, the interference of the lighter VLQ with its heavier counterpart is destructive, while for $m_{ \mathrm VLQ} < 0$ the exact opposite happens. Thus, the values of $y_\mathrm{VLQ}$ that can be probed are higher in the case of a positive mass for the lighter VLQ. Loosely speaking, the situation is the other way around if the sign of M is flipped. More precisely, under the change $M \rightarrow -M$, both $m_\mathrm{VLQ}$ and $y_\mathrm{VLQ}$ would change sign, which graphically means that the regions in Fig. 5 would undergo a reflection about the origin, defined by $\{ m_\mathrm{VLQ}, y_\mathrm{VLQ} \} = \{ 0,0 \}$.