LHC Discovery Potential for Non-Standard Higgs Bosons in the 3b Channel

In a variety of well motivated models, such as two Higgs Doublet Models (2HDMs) and the Minimal Supersymmetric Standard Model (MSSM), there are neutral Higgs bosons that have significantly enhanced couplings to b-quarks and tau leptons in comparison to those of the SM Higgs. These so called non-standard Higgs bosons could be copiously produced at the LHC in association with b quarks, and subsequently decay into b-quark pairs. However, this production channel suffers from large irreducible QCD backgrounds. We propose a new search strategy for non-standard neutral Higgs bosons at the 7 TeV LHC in the 3b's final state topology. We perform a simulation of the signal and backgrounds, using state of the art tools and methods for different sets of selection cuts, and conclude that neutral Higgs bosons with couplings to b-quarks of about 0.3 or larger, and masses up to 400 GeV, could be seen with a luminosity of 30 fb^{-1}. In the case of the MSSM we also discuss the complementarity between the 3b channel and the inclusive tau pair channel in exploring the supersymmetric parameter space.


Introduction
The origin of electroweak symmetry breaking, leading to the generation of mass for the quarks, leptons and the weak gauge bosons is one of the most outstanding questions in high energy physics. In the Standard Model (SM), the spontaneous breakdown of the electroweak symmetry is induced by the introduction of a scalar doublet field that acquires a non-vanishing vacuum expectation value. A physical particle appears in association with the Higgs mechanism, namely the Higgs boson. The tree-level couplings of quarks and leptons to the Higgs boson are then well defined and are proportional to the quark and lepton masses and inversely proportional to the vacuum expectation value of the Higgs field. Hence, apart from the top quark, all quarks and leptons have small couplings to the SM Higgs boson. Therefore the production cross section of a SM Higgs in association with all SM fermions, apart from the top quark, is too small to be detectable at hadron colliders.
It is very likely, however, that the electroweak symmetry breaking sector is more complicated than just a single Higgs doublet. One of the simplest extensions of the SM is the two Higgs doublet model (2HDM). In this case, quarks and leptons receive contributions to their masses coming from both Higgs doublets. If the vacuum expectation value of one of the Higgs doublets is small, its coupling to some of the quarks can be very large. Flavor physics puts however additional constraints on these extended Higgs sectors: in order to suppress large flavor-changing neutral-current (FCNC) interactions, either the coupling of one of the two Higgs doublets to fermions with a given electric charge is suppressed, or there is an alignment between the couplings of the fermions to the two Higgs doublets. The so-called Type II 2HDM belongs to the first class of models: up-type quarks and neutrinos couple to one Higgs doublet, H u , and down-type quarks and charged leptons couple to the other, H d .
The minimal supersymmetric extension of the SM (MSSM) contains two Higgs doublets. At the tree-level, the MSSM Higgs sector is a Type II 2HDM. However, fermion couplings to both Higgs doublets are induced at the loop level. Once Supersymmetry (SUSY) is broken, dangerous FCNC interactions are generated, but since they are proportional to loop-induced couplings, they tend to be suppressed. Due to the large top quark mass, the vacuum expectation value of the Higgs field that only couples to the up sector at the tree level (H u ) cannot be much smaller than the SM one. Defining tan β as the ratio of the vacuum expectation values of the two Higgs fields (tan β = H u / H d ), this implies that tan β should be of the order of or larger than one.
In Type II 2HDMs as well as in supersymmetric models, large values of tan β imply a large coupling of the b quark to the non-standard Higgs bosons, resulting in both a large production cross section of Higgs bosons in association with b quarks as well as a large branching fraction of the Higgs bosons decaying into b quarks. In the supersymmetric case, the precise value of the coupling depends not only on tan β, but also on SUSY-breaking effects. These effects can modify both the b and τ couplings to the non-standard Higgs bosons, and hence the branching ratio of these Higgs bosons decaying into b quarks and τ leptons.
Non-standard Higgs boson production at the LHC has been mainly studied through inclusive Higgs boson decays into τ leptons, since this channel has a reasonable signal-to-background ratio. Currently, the LHC experiments are setting strong bounds [1,2] on light non-standard neutral Higgs bosons at moderate or large values of tan β, surpassing the previous bounds [3,4] set by the Tevatron experiments. In this work, we shall study the associated production of a non-standard neutral Higgs boson with b quarks at the LHC, with the Higgs boson subsequently decaying into b quarks. The process involves the production of at least three b quarks in the final state and, for large values of tan β, the production cross section may be sizable. This search channel suffers from a large irreducible background, since the QCD bb+X production cross section is much larger than the one associated with Higgs production. Hence, previous experimental studies of this channel have been mainly performed at the Tevatron [5,6], where the backgrounds are easier to control, but no CMS or ATLAS analysis is available at present. This search channel is challenging because the b quark produced in association with the Higgs boson typically has low transverse momentum (p T ) and triggering on such soft b jets, especially for low Higgs boson masses, is difficult at high instantaneous luminosity. After satisfying the trigger requirements, demanding that two of the b jets reconstruct the Higgs boson invariant mass helps to improve the signal significance. However systematic uncertainties can still be an issue due to the small signal-to-background ratio. Previous theoretical studies focused on the prospects for the discovery of a non-standard neutral Higgs boson in the 3b and 4b channels at the 14 TeV LHC [7,8,9]. In particular supersymmetric models such as anomaly-mediated supersymmetry breaking (AMSB) models and gauge-mediated supersymmetry breaking (GMSB) models have been studied [9]. The aim of our paper is to analyze the reach at the 7 TeV LHC for Higgs bosons arising in generic 2HDMs. In particular, we determine the required effective coupling of the neutral Higgs boson to b quarks to have a possible discovery at the 7 TeV LHC with 30 fb −1 of data. We also consider the specific case of the MSSM and investigate the complementarity to the A → ττ searches.
In section 2, we shall present the necessary theoretical background and emphasize the differences between the several two Higgs doublet extensions discussed above. In section 3 we shall study the reach for non-standard neutral Higgs bosons at the 7 TeV LHC in the 3b channel. We describe our simulation of signal and background and the proposed selection cuts, and then discuss the expected reach in specific 2HDMs, as well as in the MSSM. We reserve section 4 for our conclusions and outlook.

The 3b channel in 2HDMs
In a 2HDM the most generic Yukawa couplings of the two Higgs doublets with SM quarks and leptons can be written as (1) in which H u and H d are the two Higgs doublets with hypercharge 1/2 and -1/2, respectively.
A generic structure of the four Yukawa couplings leads to Higgs-mediated FCNC interactions already at the tree level. However New Physics (NP) effects in flavor transitions can be reduced by imposing the alignment of up, down and lepton Yukawa couplings [10] or, more generically, the Minimal Flavor Violation principle [11].
We shall introduce the variables f , parameterizing the relation between the y f andỹ f couplings in alignment models, with generic flavor independent t,b,τ coefficients. In the following we will focus on the couplings of the three neutral Higgs bosons with the third-generation down-type quarks and leptons.
Assuming that there is no CP violation in the Higgs sector, the couplings of the neutral Higgs bosons with b quarks are given by [12] wherem b is the running b quark mass and α is the mixing angle between the two scalars h, H. The corresponding couplings of the Higgs bosons with the third-generation charged leptons are obtained with the simple exchange b ↔ τ . Hence, in generic aligned 2HDMs the couplings of the pseudoscalar Higgs boson with b quarks and τ leptons can be parametrized by two independent effective couplings tan β b eff and tan β τ Moreover, in the decoupling limit, arising at large values of m A and tan β, cos α ∼ sin β, sin α ∼ − cos β and consequently the coupling of the heavy scalar H with b quarks (τ leptons) is also governed by tan β b eff (tan β τ eff ). The coupling of the light scalar h is instead SM-like in this limit.
The total production rate of b quarks and τ pairs mediated by the production of a CP-odd Higgs boson (as well as by the heaviest CP-even Higgs scalar) in the large tan β regime can be approximated by [13] where N c is the number of colors (N c = 3) and σ(bbh) SM and σ(gg, bb → h) SM denote the values of the corresponding SM Higgs boson production cross sections for a Higgs boson of equal mass 1 .
The MSSM at the tree level is a particular 2HDM of Type II, however at the one-loop level also the Yukawa couplingsỹ u ,ỹ d ,ỹ are generated. In particular the dominant threshold corrections to the b quark mass are arising from gluino-sbottom one-loop diagrams and from chargino-stop loops, resulting in in which Mg is the gluino mass, Mb i and Mt i the sbottom and stop masses and µ the Higgsino mass parameter. A t and y t are the top trilinear term and the top Yukawa coupling, respectively.
Similarly, the corrections to the τ mass are dominated by wino and bino exchange contributions that are usually small, since they are suppressed by the electroweak coupling and have the form [19,20] with M 2 the wino mass, Mν τ the sneutrino mass. The effective couplings of the CP-odd and heavier CP-even Higgs bosons of the MSSM with b quarks and τ leptons are then given by Eqs. (4) and (5) with the resummation factors b and τ given just above. As a result, contrary to generic aligned 2HDMs, in the MSSM the two couplings will be uniquely determined, once the supersymmetric spectrum is specified.
3 Early LHC prospects for the 3b channel 3

.1 Simulation of Signal and Background
Signal and background processes are modeled using the MadEvent5 [21] event generator interfaced with Pythia 6.4 [22] for parton showering and hadronization, using a matrix-element parton-shower matching algorithm to avoid double-counting of partonic configurations. We allowed for up to two additional partons in the final state of the hard process when following the shower-k T scheme outlined in Refs. [23,24], with k T -matching scale of 30 GeV. These samples were generated for pp collisions at √ s = 7 TeV using the CTEQ6L1 parton distribution functions (PDFs) [25].
The QCD production of multiple heavy quarks is the main source of background. We generated two separate QCD background samples: a bbj + X (j = u, d, s, c, g) sample in which the additional partons used in matching to the parton-shower are light or charm quarks and a "3b" (bbb + X and bbb + X) sample in which the additional partons could be either light or heavy quarks. 2 We cluster particle jets using the anti-k T algorithm implemented in FastJet-2.4.3 [26] with a radius parameter R = 0.4. To better simulate the experimental b jet energy resolution, we exclude neutrinos from jet clustering, which in the case of semileptonic b decays can carry away a significant fraction of energy. Furthermore, we apply a jet energy smearing of 100%/ E/GeV to model the typical calorimeter energy resolution of LHC experiments.  Since the signal typically contains three b quarks, flavor tagging becomes an effective tool to suppress QCD multijet backgrounds. We assume a constant b-tagging efficiency of 60%, a c-jet mis-tag rate of 10% and a light-jet mis-tag rate of 1% [27]. This choice can be considered conservative, as LHC experiments have already developed sophisticated b-tagging algorithms [28] exceeding the performance assumed in this paper. The low mis-tag rate of cand light-jets leads to the bbj and 3b backgrounds being comparable once three b-tagged jets are required.
We consider two sets of event selection criteria: 1. Selection I: events are required to have exactly three b-tagged jets with p T > 60 GeV and |η| < 2.0.
2. Selection II: events are required to have exactly three b-tagged jets with p T > 50 GeV and |η| < 2.0, and the leading b-tagged jet to have p T > 130 GeV.
In both cases jets are required to be relative central to ensure they are contained within the tracker volume and can therefore be tagged with high efficiency [27].
The high instantaneous luminosities delivered by the LHC has forced to raise threshold in the trigger menus in ATLAS and CMS. As a result, the first selection is somewhat optimistic in that the jet p T requirements may be too low for these events to satisfy trigger requirements with high efficiency. For instance, events satisfying Selection I would have a low efficiency to satisfy the 2b/2j ATLAS trigger [29] requirements, which appears as one of the most suitable unprescaled triggers for this topology. On the other hand, Selection II would be more representative of the kind of minimum jet p T requirements applied by the 2b/2j ATLAS trigger once trigger turn-on effects are considered. Nevertheless, we consider Selection I to explore the potential sensitivity gains at low m A values, which could motivate designing an optimized trigger strategy for such lower -p T events at the LHC experiments.
In most part of the parameter space under consideration, there is no obvious way to accurately distinguish the pair of b jets coming from the decay of the heavy scalar and the b jet produced in association with it. However Selection II with masses m A ∼ < 260 GeV represents an exception: the b jet produced in association with the Higgs boson is the leading one (b 1 ) because the b jets coming from the decay of the Higgs are too soft to satisfy the selection criteria. Therefore in Selection II for low m A the invariant mass of the second-leading b jet (b 2 ) and the third-leading b jet (b 3 ) reconstructs the Higgs mass, while for large m A (m A ∼ > 260 GeV) the invariant mass of b 1 with b 2 or b 3 reconstructs the Higgs mass. Nevertheless to improve the acceptance, we consider all three possible combinations of b-jet pairs and require that the invariant mass of at least one of them is within a window around the peak of the Higgs boson invariant mass distribution. Figure 1 compares the invariant mass distribution between signal and background for each of the possible b-jet pairs. As it can be appreciated, the signal distribution consistently peaks at values lower than the physical mass of the Higgs boson due to a combination of the jet smearing 3 and energy loss via neutrinos, with the PDF suppression involved in producing such a massive resonances. This effect becomes more significant for heavier resonance. If an excess in the invariant mass distribution of a pair of b quarks is discovered at the LHC, one needs to extract the actual mass of the resonance through proper simulation. The chosen central value of the mass window is shown in Table 1 for each of the m A values considered.
Additionally, we studied the effect on varying the width of the mass window about the peak and found that a typical width of |M bb − m peak | ≤ 25 GeV yields good results across the whole mass range. Increasing the width of the mass window for heavier masses and reducing the width for lower masses can lead to a improvement in the significance of only of a few percent. Finally, we checked that imposing a cut on ∆R between any two b jets did not lead to a marked improvement in the signal significance.  Table 1: Physical masses, m A , used in our signal samples, and the corresponding central values of the mass window. The mass window used in the signal selection is chosen to be |M bb − m peak | ≤ 25 GeV.

Prospects and Significance
As discussed in Sec. 2, in a generic 2HDM the couplings of the A, H bosons with b quarks and with τ leptons are independent and parametrized by the effective couplings tan β b eff and tan β τ eff , respectively. As shown by Eq. (6), the cross section times branching ratio for nonstandard neutral Higgs bosons produced in association with a b quark which subsequently decay into a pair of b quarks has only a mild dependence on the choice of tan β τ eff . Without lost of generality we fix tan β τ eff = 5 to be in agreement with the present bounds coming from LHC A, H → ττ searches [1,2].
In addition, we are focusing on the parameter region with sizable tan β b eff and moderate values of m A . In this case, the heavy CP-even scalar H and the pseudoscalar A can only be slightly split in mass, so that the two Higgs bosons will appear at the LHC in the same resonance region with combined cross section. Hence, for our analysis, the only two relevant free parameters are m A ∼ m H and tan β b eff . To compute the rate of the signal, we double the cross section obtained for the CP-odd Higgs and use the narrow width approximation which is valid in the entire mass range we consider as long as tan β b eff is not too large (tan β b eff 80).    Table 3: Expected number of background and signal (at tan β b eff = 30) events per 30 fb −1 of data at the 7 TeV LHC, after imposing Selection II presented in the text (above double line) and after the mass window selection presented in Table 1 (below double line). The first row shows the total events in each channel before event selection criteria are imposed.
In Tables 2 and 3 we present the number of signal and background events per 30 fb −1 in each of the test mass windows, assuming tan β b eff = 30. In Table 4, we compare the signal statistical local significance 4 from applying Selection I with that of Selection II, assuming an integrated luminosity of 30 fb −1 . We can see that, as expected, Selection I has a markedly better statistical sensitivity for m A < 300 GeV.
Using Eq. (6), it is straightforward to generalize these results to different values of tan β b eff . In Fig. 2 we present the accepted signal cross section, after that all cuts are implemented, and the statistical significance at 30 fb −1 LHC as a function of the two free parameters of the theory, tan β b eff and m A for both Selection I and Selection II. The results are encouraging. For a 7 TeV LHC run with a total integrated luminosity of 30 fb −1 , we can probe a large parameter region of the 2HDM by searching for heavy Higgs scalars in the 3b final state. For example, applying Selection I, a coupling of the pseudoscalar with b quarks of the order ∼ 0.3, and hence tan β b eff ∼ 30, could lead to a 5σ significance for a Higgs boson mass up to ∼ 370 GeV, with 30 fb −1 of data. However, from Table 4, we also see that for moderate tan β b eff , the final S/B 0.1. Therefore the potential systematic uncertainties which have not been accounted for in our analysis could make signal identifications challenging. We expect that a detailed experimental analysis exploiting the sideband regions in the invariant mass distribution to constrain systematic uncertainties in the background prediction can nevertheless achieve a high sensitivity. In any case, for larger values of tan β b eff (∼ 60), and hence a coupling ∼ 0.6, we can achieve a significance ∼ > 10 with 30 fb −1 of data in almost the entire range of masses considered. For such large effective couplings also the signal-to-background ratio would be more favorable, S/B 1/5.
Comparing the first and second rows of Fig. 2, we note that there is a difference in shape of the contours. Applying Selection II, the best reach is for m A ∼ 250 GeV. On the other hand, the reach when using Selection I monotonically decreases as the value of m A increases, which is the result of the rapid decrease in signal rate. This difference in the shapes of the accepted signal cross section and of the statistical significances are related to the different cuts on the three highest p T jets. First of all, the b jet produced in association with the Higgs boson has a steeply falling distribution, suppressed by the PDFs. Therefore, it is unlikely that this jet  Table 1. In the bottom row, the results from applying Selection II and the mass window cuts presented in Table 1.
can satisfy the cut of p T > 130 GeV on the leading jet in Selection II. This explains why Selection I leads to better sensitivity, in particular in the low mass region. On the other hand, for m A ∼ 250 GeV, it is easier for the b jets from Higgs decay to be the leading jet and to satisfy this cut. This explains why the reach in Selection II is better for higher Higgs masses than for lower ones. This effect suggests that an asymmetric jet energy cut, similar to that of Selection II, could have advantages. The p T cut on the leading jet could be optimized further, such as requiring it to be proportional to the target signal mass.
The possibility of detecting the pseudoscalar and the heavy scalar of the MSSM in the 3b channel deserves a special discussion, since in the MSSM the effective couplings tan β b eff and tan β τ eff defined in Eqs. (4) and (5) are determined, once the SUSY spectrum is specified. As discussed in Sec. 2, contrary to generic 2HDMs, in the MSSM the coupling of the pseudoscalar Higgs with b quarks and τ leptons depend equally on tan β but have a different dependence on corrections arising at the one-loop level. Typically, for gluinos at the TeV scale, stops, sbottoms and charginos at a few hundred GeV and A t of the order 1-2 TeV, b is at the few % level. On the other hand, in the lepton sector typically τ ∼ O(10 −3 ). 5 In our numerical analysis, we choose two representative scenarios: the first with τ = 0 and b = −1/60, and the second with τ = 0 and b = −1/30. Both scenarios can be achieved in models with a large and negative µ term (see Eqs. (8) and (9)). The effects of introducing a small but non-zero τ will not significantly modify our conclusions. These scenarios are presented in Fig. 3. The plots on the left represent the case τ = 0 and b = −1/60; the ones on the right τ = 0 and b = −1/30. In white we present the bound on tan β coming from the requirement that the narrow width approximation is valid (Γ A m A 10 ). It is interesting to compare the shape of the exclusion bound from the LHC A → ττ search to the one of the constant significance contours for the A → bb channel. For large values of m A , the present CMS bound [2] obtained with ∼ 5 fb −1 of data is weaker than what was expected. As a consequence, the slope of the present ττ exclusion bound (solid red line in the figure) is much steeper than the slope of the A → bb constant significance contours, at large values of m A . Differently, the slope of the expected bound projected at 30 fb −1 (dashed red line in the figure) gets much closer to the slope of the A → bb constant significance contours, especially at small values of b ( b = −1/60). For larger b ( b = −1/30) one can still note a difference in the slopes, that is mainly due to the different scaling in tan β of pp → bbA with A → bb and pp → bbA with A → ττ , as shown by Eqs. (6), (7) once tan β b eff and tan β τ eff are replaced with their expressions in (4), (5) and τ is set to 0. Thanks to this different scaling, the 3b channel can be used to probe heavier pseudoscalar masses than the ττ channel. Whether the 2σ excess at high mass from CMS turns out to be a hint for a signal or just the result of a statistical fluctuation, it seems imperative to probe this mass range with a channel with comparable or better sensitivity, able to provide complementary information on the MSSM preferred region of parameter space.

Conclusions
In this article we have studied the associated production of non-standard neutral Higgs bosons with b quarks at the LHC. Considering the Higgs boson decay into b quarks, we have analyzed the reach of the 7 TeV LHC collider. After applying a rather simple event selection cuts, a manageable signal-to-background ratio could be obtained, helped by the enhanced production cross section obtained at large values of tan β, which allows for a statistically meaningful reach at moderate values of the LHC luminosity. In particular, we find that Higgs bosons with a coupling to b quarks of about 0.3 or larger (namely tan β b eff = 30 or larger), and with a mass up to 400 GeV could be discovered with a luminosity of 30 fb −1 . We expect that the run at 8 TeV will enhance the reach by about 10-15%, although a precise estimation of the reach depends on many details which are beyond the scope of this study.
We have studied the discovery potential using two different sets of cuts. In general, the b jet produced in association with the Higgs boson tends to be soft, driven by the suppression from the steeply falling PDFs. At the same time, the p T of the b jets from the Higgs decay is closely correlated to the mass of the Higgs boson. Therefore, in particular in the low mass region, a somewhat lower threshold on the total jet p T will enhance the discovery reach. At the same time, it could be beneficial to use an asymmetric p T selection criteria with the requirement that the cut on hardest b-jet p T is correlated with the target Higgs mass. This effect should be more prominent for higher Higgs masses.
We have also studied the discovery potential in SUSY-like scenarios. In this case, the corrections to the Yukawa couplings arise at loop level, and there is a correlation between the bb and ττ search channels. We found that the 3b channel can be important in probing supersymmetric scenarios in which SUSY-breaking effects can significantly modify the couplings of non-standard neutral Higgs bosons to b quarks and τ leptons. In particular we showed that the ττ channel still has a better reach for lower Higgs boson masses, but the bb channel can be used to probe heavier pseudoscalar masses than the ττ channel. Furthermore, the 3b channel provides an important probe into the coupling of the Higgs boson to b quarks and hence it is complementary to the ττ channel.