Searching for a charged Higgs boson with both $H^{\pm}W^{\mp}Z$ and $H^{\pm}tb$ couplings at the LHC

In certain new physics scenarios, a singly charged Higgs boson can couple to both fermions and $W^\pm Z$ at tree level. We develop new strategies beyond current experimental searches using $pp\to jjH^\pm$, $H^\pm \to tb $ at the Large Hadron Collider (LHC). With the effective $H^\pm W^\mp Z$ and $H^\pm tb$ couplings we perform a model-independent analysis at the collision energy $\sqrt{s}=13$~TeV with the integrated luminosity of $3~\text{ab}^{-1}$. We derive the discovery prospects and exclusion limits for the charged Higgs boson in the mass range from 200~GeV to 1~TeV. With $|F_{WZ}|,|A_t|\sim 0.5-1.0$ and $300~\text{GeV}\lesssim m_{H^\pm}\lesssim 400~\text{GeV}$, we point out that a discovery significance of $5\sigma$ can be achieved. The constraints and projected sensitivities are also discussed in a realistic model, i.e., the modified Georgi-Machacek model without custodial symmetry. Our proposed search would provide direct evidence for a charged Higgs boson $H^\pm$ that couples to $W^\pm Z$ and $tb$, which can have better sensitivity to the couplings of $H^\pm W^\mp Z$ and $H^\pm tb$ than current searches.


I. INTRODUCTION
(GM) model, in which the triplet VEV can be large. There are two singly charged Higgs bosons H ± 3 and H ± 5 in the GM model, which transform as 3-plet and 5-plet under SU (2) C , respectively [21]. Due to the custodial symmetry, H ± 3 and H ± 5 couple to fermions and W ∓ Z separately and there is no mixing between them. Besides, both the couplings of H ± 3 to fermions and H ± 5 to W ∓ Z are proportional to the triplet VEV [23,24]. Experimental searches for singly charged Higgs bosons have been carried out at highenergy colliders. The search strategies depend on the couplings of charged Higgs boson to SM particles. The LEP experiments have excluded the charged Higgs boson in the type-II 2HDM with mass below 80 GeV [25]. At the LHC, the H ± D in the 2HDMs is primarily produced from top quark decay when lighter than top quark, i.e. m H + D < m t or in the association production with top quark if m H + D > m t . Various decay channels including 3 tb [26][27][28] τ ν [28-31], cs [32] and cb [33] have been explored for H ± D in the mass range from 90 GeV to 2 TeV with no significant excess observed. For the H ± 5 in the GM model, it has been sought in the vector-boson-fusion (VBF) process pp → jjH ± 5 , H ± 5 → W ± Z [34,35] at the LHC for 200 GeV ≤ m H ± 5 ≤ 1 TeV. However, an interesting scenario in which H ± couples to both fermions and W ∓ Z significantly has not been extensively explored. Such a charged Higgs boson does not exist in many popular Higgs extended models, such as 2HDMs and GM model. Nevertheless, once a singly charged Higgs boson was discovered in one existing channel, its couplings to all SM particles are requested to verify its nature. In this scenario, new production and decay channels can be utilized. Since the couplings of H ± to fermions are usually proportional to the fermion masses, its couplings to the third generation quarks are more relevant. The property of charged Higgs boson with both couplings to fermions and W ∓ Z can be revealed through the processes: (1) pp → jjH ± , H ± → tb; (2) pp → tH ± , H ± → W ± Z. In this work, we will study the first process pp → jjH ± , H ± → tb by investigating the sensitivity to the effective couplings of H ± to W ∓ Z and tb and discussing the implications in a realistic modified GM model [36,37] 4 Whereas, we emphasize that our results are model-independent and can be applied to other models with such charged Higgs boson(s). The second process is left for a future work This paper is arranged as follows. In Section II, the effective H ± W ∓ Z and H ± tb couplings and decay branching ratios are discussed. With the effective operators, we perform a model-independent collider study with the currently available constraints and the projected sensitivity in our proposed search pp → jjH ± , H ± → tb. In Section III, we discuss the search of a charged Higgs in the modified GM model and the implications on model parameters. Section IV summarizes our results.
II. H ± W ∓ Z AND H ± tb COUPLINGS AND COLLIDER SEARCH STRATEGIES The most general effective Lagrangian that describes the interactions of H ± with W ± Z and tb is parameterized as [40,41] where P L/R = (1 ∓ γ 5 )/2, and v 246 GeV. For m H + > m t + m b , the partial widths of H + into W + Z and tb are given by Here g is SU (2) L gauge coupling, λ(x, y, z) = (x−y−z) 2 −4yz and x W,Z,t,b = m 2 W,Z,t,b /m 2 H + . The H + has the total width where Γ(H + → others) denotes the partial width of H + into other final states. Since the couplings of H + to fermions are proportional to the fermion masses, the decay widths of H + into other fermions are much smaller. Thus it is plausible to assume that the total width of H + is saturated by decays into W + Z and tb (minimal total width assumption). Decay branching ratios of H ± → tb and H ± → W ± Z are expressed as In the following, the effective couplings in Eq.
(1) will be adopted. We perform a model-independent analysis of the pp → jjH ± , H ± → tb with leptonic decay of top quarks at the 13 TeV LHC. The charged Higgs boson mass is assumed to be in the range from 200 GeV to 1 TeV 5 . We note that in Ref. [42], the process pp → jjH ± , H ± → tb was also studied at the LHC. Only the effective coupling H ± W ∓ Z was discussed, whereas how H ± can simultaneously decay into tb was not explained [42]. We will perform a more detailed collider analysis in this section and investigate a realistic model, which has both tree-level couplings of H ± W ∓ Z and H ± tb in Section III.
5 We choose this mass range to match the current searches in the VBF channels in Refs. [34,35].
In the above, m, n = j, b, , j denotes light-flavor quarks, = e, µ, and the angular distance in the η − φ plane is defined as ∆R ij ≡ (η i − η j ) 2 + (φ i − φ j ) 2 with η i and φ i being the pseudo-rapidity and azimuthal angle of particle i, respectively. The NN23LO1 Parton Distribution Function (PDF) set [44] and default hadronization and factorization scales are used. The parton-level events are interfaced to Pythia 6.4 [45] and Delphes3 v3.3.3 [46] for parton shower and detector simulation. The backgrounds tt, tW , tq are matched in 5-flavor scheme up to 1, 1 and 2 jets, respectively. Jets are clustered via the anti-k t algorithm [47] with a radius parameter of R = 0.4 as implemented in Fastjet [48].
To select the signal process, we impose a series of selection cuts, which are composed of basic, VBF and optimal cuts. The basic cuts are used to identify objects at the hadron level: (B-1) The angular separation is chosen as ∆R mn > 0.4, m, n = j, b, . (B-4) It is required that the missing energy E miss T > 30 GeV [50] since there is a neutrino of the signal process in the final state.
The VBF cuts are adopted similar with those in Ref. [34,35]: (V-1) There are at least two non-b-tagged jets in opposite detector hemispheres.
(V-2) The invariant mass and rapidity separation of the leading p T jets in opposite hemispheres are used as m jj > 400 GeV, |∆η jj | > 3.5.
High-p T b-jets and charged lepton from the decay H ± → tb, t → b ν can be further used to trigger the signal. Besides, since there is only one neutrino in the signal process, it is possible to fully reconstruct the four-momentum of top quark and the invariant mass of the charged Higgs boson H ± 's decay products. Depending on the kinematic distributions of p T , p b1 T , p b2 T and m tb after the VBF cuts for each m H ± (see Fig. 1 for distributions with m H ± = 500 GeV), one can impose the optimal cuts as follows: (O-1) The transverse momentum of charged lepton is larger than a minimum, p T > p T min .
(O-2) Lower bounds or upper bounds of the transverse momenta of tagged b-jets are imposed depending on m H ± , T and m tb after the VBF cuts for the charged Higgs boson mass m H ± = 500 GeV. Here "HP500" and "ttxj", "twj", "tqjj" denote the signal and backgrounds tt, tW , tq, respectively.
where p b1 T and p b2 T denote the leading and sub-leading transverse momenta of b-jets.
(O-3) The four-momentum of top quark can be reconstructed using the template χ 2 method. For each event, the χ 2 is defined as [51] where m t = 172.5 GeV [52], Γ t = 4.08 GeV, m b ν denotes the invariant mass of lepton, neutrino and one b-jet. With the on-shell condition of W boson, we can determine the longitudinal transverse momentum of neutrino with a two-fold ambiguity [53] where A W = m 2 W + 2 p T · p miss T and m W = 80.4 GeV. Furthermore, there are two b-jets in the final state 6 . By minimizing χ 2 [54], we can determine which b-jet originates from top quark decay and the sign of p ν L . Thus one is able to fully reconstruct the four-momentum of top quark and the invariant mass of t and b (m tb ). Since in the signal process, t and b come from the decay of on-shell charged Higgs boson H ± , the variable m tb can be used to suppress the backgrounds. The cut on m tb is optimized depending on m H ± , Here, p T min denotes the lower bound of p T . Similarly, p b1 T min/max , p b2 T min/max and m min/max tb are the lower/upper bounds of p b1 T , p b2 T and m tb , respectively. The explicit values are given in Tab. I.
After imposing the selection cuts, we find that the significance of the signal can be greatly improved. We show in Tab. II the cut flow of the signal and background cross sections after each selection cut for m H ± = 500 GeV with the benchmark scenario F W Z = A t = 0.5 . The discovery prospect and exclusion limit of the signal process are evaluated using [55] respectively. n s = σ s L and n b = σ b L denote the number of events after cuts with the integrated luminosity of L, σ s and σ b are the cross sections of the signal and total background after cuts. The signal cross section σ s can be expressed as where the quantities in the square bracket are obtained with the benchmark values F W Z = A t = 0.5. The signal cut efficiency s is independent of the couplings F W Z and A t 7 , so that we are free to obtain it with the benchmark values of F W Z and A t . The signal cross section is proportional to |F W Z | 2 × B tb , which depends on |F W Z | and |A t | as shown in Fig. 2. We find that |F W Z | 2 × B tb tends to be larger for a smaller m H ± . At the 13-14 TeV LHC, the integrated luminosity is planned to reach 3 ab −1 [56]. In this work, we will study the process pp → jjH ± , H ± → tb with integrated luminosity  of 3 ab −1 at the 13 TeV LHC. Using the relation in Eqs. (9) (10) (11), we can obtain sensitivity to |F W Z | 2 × B(H ± → tb) from the discovery prospects and exclusion limit of our proposed signal process pp → jjH ± , H ± → tb as shown in Fig. 3. From this figure, one can find that, the scenarios with |F W Z | 2 ×B(H ± → tb) 0.43, 0.72 can be observed at 3σ, 5σ for the charged Higgs boson in the mass range from 200 GeV to 1 TeV, respectively. While the region of |F W Z | 2 × B(H ± → tb) 0.25 would be excluded at 95% confidence level (C.L.) or equivalently with Z E ≥ 1.96. It is interesting to note that the sensitivity is the best for m H ± 250 GeV in the process pp → jjH ± , H ± → tb.
On the other hand, there are already experimental searches for charged Higgs bosons in the 2HDMs and the GM model. They provided constraints on |A t | 2 × B(H ± → tb) and |F W Z | 2 × B(H ± → W ± Z), respectively. To recast current constraints, we generate the leading-order (LO) processes pp →tH ± and pp → jjH ± in 5-flavor scheme using MG5 aMC@NLO v2.4.3 with F W Z = A t = 0.5. We denote the exclusion limits at 95% C.L. in Refs. [26-28, 34, 35] of σ(pp → tH ± )B(H ± → tb) and σ(pp → jjH ± )B(H ± → W ± Z) as [σB] limit tb and [σB] limit W Z , respectively. The upper limits can then be re-interpreted as where σ(pp → tH ± ) BM and σ(pp → jjH ± ) BM denote the LO cross sections 8 of the pp → tH ± and the pp → jjH ± , respectively. The constraints on |F W Z | 2 ×B(H ± → W ± Z) and |A t | 2 × B(H ± → tb) are shown in Fig. 4. One can see that the constraints at the 13 TeV LHC are more stringent than those at the 8 TeV LHC, except for 300 GeV < m H ± < 450 GeV in the process pp → jjH ± , H ± → W ± Z. In the following analysis, we take the strongest constraints in the range of m H ± from 200 GeV to 1 TeV. On account of benchmark mass points in Ref. [34], we will choose the mass interval of 100 GeV to illustrate the combined sensitivities in Fig. 5 and Fig. 7. the exclusion limits at 95% C.L. of the processes pp → tH ± , H ± → tb and pp → jjH ± , H ± → W ± Z, respectively. The red solid, dashed and dot-dashed curves correspond to the exclusion limits at 95% C.L. and discovery significance Z D = 3, 5, respectively.
With the minimal total width assumption, we can express the branching ratios B(H ± → tb) and B(H ± → W ± Z) in terms of the effective couplings F W Z and A t . Therefore, we can obtain the discovery prospects and 95% C.L. exclusion limits in the plane of |F W Z | and |A t |. In Fig. 5, scenarios outside of the shaded regions are excluded at 95% C.L. by the existing searches or our proposed search. The dashed and dot-dashed curves correspond to the discovery significance of 3σ, 5σ, respectively. For brevity, the processes pp → tH ± , H ± → tb and pp → jjH ± , H ± → W ± Z are denoted as tb and W Z modes, respectively. The process pp → jjH ± , H ± → tb that depends on both the H ± W ∓ Z and H ± tb couplings is denoted as the mixed mode. We find that for 300 GeV m H ± 400 GeV with |F W Z |, |A t | ∼ 0.5 − 1.0, H ± that couples to W ± Z and tb can be discovered in the process pp → jjH ± , H ± → tb at Z D ≥ 5. For lighter or heavier charged Higgs boson, we can still achieve regions of the effective couplings that correspond to Z D ≥ 3. If m H ± ≥ 500 GeV, top quark from the decay of H ± is boosted. It is possible to improve the significance with jet substructure techniques [27,60], which is beyond the scope of this study. On the other hand, if a null result is observed, 95% C.L. exclusion limits in the plane of |F W Z | and |A t | are imposed. We obtain the most sensitive constraints on models with both H ± W ∓ Z and H ± tb couplings from the process pp → jjH ± , H ± → tb for |F W Z |, |A t | ∼ 1.0.

III. MODIFIED GM MODEL WITH BOTH H ± W ∓ Z AND H ± tb COUPLINGS
In this section, we will investigate a realistic model with nonvanishing H ± W ∓ Z and H ± tb couplings. Recall that in the GM model [20,21], there are two singly charged Higgs boson H ± 3 and H ± 5 , which interact with fermions and W ∓ Z separately since they transform in different custodial symmetry SU (2) C representations. It was found [36,61] that if the custodial symmetry in the Higgs potential is relaxed, these two charged Higgs bosons can in general mix with each, resulting in two mass eigenstates that couple to both W ± Z and tb. We will refer this model as the modified GM model. However, we emphasize that it is just a representative model. For example, in a supersymmetric model [62,63], charged Higgs bosons can also couple to W ± Z and fermions simultaneously due to the custodial symmetry breaking and doublet-triplet mixing. Our results in Section II can be applied to any model with H ± W ∓ Z and H ± tb interactions. Below we will briefly introduce the modified GM model and show the implications in terms of the model parameters.
As in the original GM model [20,21], we introduce two Higgs triplets: one real triplet ξ ∼ (3, 0) and one complex triplet χ ∼ (3, 1), which have the following forms: with the conventions ξ − = (ξ + ) * , χ 0 = (v χ + h χ + iI χ )/ √ 2 and ξ 0 = v ξ + h ξ . In order to discuss mass eigenstates of charged Higgs fields, it is convenient to use the basis (G + , H + 3 , H + 5 ), where v ≡ v 2 H + 4v 2 ξ + 2v 2 χ 246 GeV and the constants N i are given by The Goldstone mode G + is "eaten" by W + , while H + 3 and H + 5 are not mass eigenstates in general unless there is custodial symmetry in the Higgs potential. One needs to further diagonalize the mass matrix to obtain the mass eigenstates H m+ 3 and H m+ 5 , The explicit form of the mixing angle δ can be found in Ref. [36]. The interactions of H m± 3 , H m± 5 with W ∓ Z and quarks are where c W ≡ cos θ W , U = (u, c, t) T , D = (d, s, b) T and V CKM denotes the Cabibbo-Kobayashi-Maskawa matrix. The ρ parameter is expressed as With custodial symmetry, v ξ = v χ / √ 2 and sin δ = 0, thus ρ = 1 and H m± 5 (= H ± 5 ) does not couple to quarks, whereas H m± 3 (= H ± 3 ) does not couple to W ∓ Z at tree level. Since the ρ parameter is severely constrained, ρ exp = 1.00039 ± 0.00017 [52], the v χ , v ξ are generally constrained to be less than a few GeV if they are independent. However when v ξ = v χ / √ 2, one has ρ = 1, and larger v χ and v ξ are allowed. In this case, tan 2δ is proportional to the combination 2v , where µ χHH , µ ξHH , κ 3 and λ are the couplings in the general Higgs potential [36] 9 . In the GM 9 If the custodial symmetry is removed, these couplings are independent, while in the GM model, µ ξHH = √ 2µ χHH and κ 3 = − √ 2λ 5 [36].
model this combination is forced to be zero by the custodial symmetry. However, if the custodial symmetry is broken explicitly the mixing angle δ can be sizable. We will work in the modified GM (MGM) model with the working hypothesis v ξ = v χ / √ 2. The interactions of charged Higgs bosons to quarks and W ± Z in the MGM model can be expressed as where The mixing angle δ is in the range of [0, 2π), so H m± 3 and H m± 5 can interact with both quarks and W ± Z unless δ = nπ/2 with n = 0, 1, 2, 3.
Compared with the Lagrangian in Eq.
(1), we can obtain the effective couplings F W Z , A t and A b as follows: for the H m± 3 and for the H m± 5 . Combined with the effective Lagrangian in Eq. (1), it is apparent that the magnitudes of the right-handed H ± tb couplings are suppressed and smaller than the left-handed one by a factor of m b /m t .
The couplings of H m ±

3
and H m ± 5 to quarks and W ∓ Z are proportional to the triplet VEV v χ , which is a generic feature of the GM-type models [64]. A larger triplet VEV can result in a better sensitivity. With the sum rule v 2 H + 4v 2 χ = v 2 in the MGM model, the triplet VEV v χ = v 2 − v 2 H /2 88, 121, 123 GeV is required in order to satisfy the perturbative bound of the top Yukawa coupling m t /v H 1, √ 4π, 4π from the renormalization group running [65,66]. Moreover, in general we can obtain a strict upper bound on the triplet VEV from the perturbative unitarity of the scattering amplitude for tt → tt [67,68]. One can obtain that y t 16π/5, which implies that the doublet VEV v H 78 GeV. In the MGM model, the couplings of the neutral Higgs bosons to tt also depend on other parameters compared with the model discussed in Refs. [67,68]. But the constrain can still provide some guide for the allowed Yukawa coupling, so that the triplet VEV v χ 117 GeV.
On the other hand, the H m± 3 tb and H m± 5 tb couplings can also modify the Zbb coupling through higher order corrections. At the 1-loop level, the correction to the left-handed coupling is expressed as [61,69]  regions are obtained from the exclusion limit at 95% C.L. of the processes pp → tH ± , H ± → tb and pp → jjH ± , H ± → W ± Z, respectively. The red solid, dashed and dot-dashed curves correspond to the exclusion limit at 95% C.L. and discovery prospects with Z D = 3, 5, respectively. The cyan vertical regions on the right of the panels are excluded by the perturbative unitarity requirement.
The theoretical R b in the MGM model is expressed as where the SM value R SM b = 0.2158 [52], δR MGM b that depends on δg L MGM can be parame-terized as [61] δR only depends on v χ and sin δ. The 1σ and 2σ constraints from R b measurements are implemented by requiring Here the subscripts "cent" and "err" denote the central value and error of R exp b in Eq. (25). In Fig. 6, we show the 1σ, 2σ contours in the plane of v χ and sin δ for m H m+ . We find that v χ 100 GeV is still allowed by the perturbative bound and the Zbb data, while the exact upper limit depends on the mixing angle δ and the interplay of two charged Higgs bosons H m± 3 and H m± 5 . Therefore, we do not show the Zbb constraint explicitly in the sensitivity plots in Fig. 7 and Fig. 8. In Section II, we have obtained the model-independent sensitivities to the effective H ± W ∓ Z and H ± tb couplings. Having the explicit forms of F W Z and A t in the MGM model in Eqs. (22) (23), we can obtain the sensitivities to the model parameters. i.e., the mixing parameter δ and the triplet VEV v χ . We assume that the charged Higgs boson H ± is identified as H m± 5 , and it mainly decays into W ± Z and tb. In the mass range from 200 GeV to 1 TeV, sensitivities to v χ and | sin δ| are shown in Fig. 7. In accord with Fig. 5, the charged Higgs boson H m ± 5 can be discovered (Z D ≥ 5) in the process pp → jjH ± , H ± → tb for 300 GeV m H m± 5 400 GeV with the triplet VEV 80 GeV v χ 100 GeV.
On the other hand, if the H m ± 5 is not found, one can put 95% C.L. exclusion limits of the signal process in the plane of v χ and | sin δ|. Especially, v χ 53, 62 GeV for | sin δ| 0.7 can be achieved with m H m± 5 = 300, 400 GeV, which are well below the upper limits 90, 80 GeV for | sin δ| = 0, respectively. To evaluate the sensitivities to the triplet VEV as a function of the charged Higgs boson mass. We choose that | sin δ| = 1/ √ 2, and obtain the sensitivities in the plane of m H m± 5 and v χ in Fig. 8. The current constraint obtained from the exclusion limit in the process pp → tH ± , H ± → tb is shown in the green curve, while there is no useful constraint from the process pp → jjH ± , H ± → W ± Z. The 3σ, 5σ discovery prospects and exclusion limit at 95% C.L. in the process pp → jjH ± , H ± → tb are also shown. Both of the processes pp → jjH ± , H ± → tb and pp → tH ± , H ± → tb get the best sensitivities for m H m± 5 250 GeV resulting dips in the curves. We obtain that the sensitivity to the triplet VEV v χ of our proposed search is better than that of the existing searches. For the m H m± 5 500 GeV, the region of the triplet VEV v χ 80 GeV can be probed at 95% C.L., shown as the purple curve in Fig. 8.
Before enclosing this section, we emphasize that the MGM model is only a representative model. Other models with triplets or higher SU (2) L representation of Higgs multiplets, in which charged Higgs boson(s) can couple to fermions and W ± Z simultaneously, can be studied similarly.

IV. CONCLUSIONS
In this work, we have extended the existing searches of charged Higgs boson at the LHC. Different from the processes inspired by the 2HDMs and GM model, the VBF process pp → jjH ± , H ± → tb requires the existence of both H ± W ∓ Z and H ± tb couplings. We have performed a model-independent analysis of this process at the LHC with the effective H ± W ∓ Z and H ± tb couplings. With the minimal total width assumption, we interpret the results in terms of the effective couplings F W Z and A t for m H ± in the range from 200 GeV to 1 TeV. We found that the process pp → jjH ± , H ± → tb can be discovered (Z D ≥ 5) for 300 GeV m H ± 400 GeV with |F W Z |, |A t | ∼ 0.5 − 1.0. Discovering the process in the region of m H ± ≥ 500 GeV requires the improved experimental selection criteria. However, one can still obtain the most sensitive constraints on models with both H ± W ∓ Z and H ± tb couplings through this process.
We investigate the implications in a realistic model, the MGM model, which introduces two Higgs triplets into the SM analogous to the GM model. Since the requirement of custodial symmetry in the Higgs potential after the EWSB is relaxed, two physical singly charged Higgs bosons H m ± 3 and H m ± 5 with both couplings to quarks and W ± Z are achieved. We discuss the theoretical as well as experimental constraints of this model. Then the sensitivities to the model parameters, i.e., the triplet VEV v χ and the mixing angle δ are obtained and compared with constraints from the existing searches if the charged Higgs boson H ± is identified as H m± 5 . We have pointed out that H m ± 5 can be discovered in the process pp → jjH ± , H ± → tb for 300 GeV m H m± 5 400 GeV with 80 GeV v χ 100 GeV. Supposing a maximal mixing pattern with | sin δ| = 1/ √ 2, the exclusion limit at 95% C.L. on the triplet VEV v χ 80 GeV can be achieved for m H m± 5 500 GeV. Finally, the signal process proposed in this work is a direct evidence for a charged Higgs boson that couples to fermions and W ± Z, which is complementary to current searches for charged Higgs bosons. Our study in this work can be used as a roadmap of future charged Higgs boson searches at the LHC.