Disentangling Flavor Violation in the Top-Higgs Sector at the LHC

We study the LHC phenomenology of flavor changing Yukawa couplings between top quark, Higgs boson, and either an up or charm quark. Such $tuh$ or $tch$ couplings arise for instance in models in which Higgs sector is extended by the existence of additional Higgs bosons or by higher dimensional operators. We emphasize the importance of anomalous single top plus Higgs production in these scenarios, in addition to the more widely studied $t \to h j$ decays. By recasting existing CMS searches in multilepton and diphoton plus lepton final states, we show that bounds on $tuh$ couplings are improved by a factor of 1.5 when single top plus Higgs production is accounted for. We also recast the CMS search for vector boson plus Higgs production into new, competitive constraints on $tuh$ and $tch$ couplings, setting the limits of $BR(t\to hu)<0.7%$ and $BR(t\to hc)<1.2%$. We then investigate the sensitivity of future searches in multilepton channel and in fully hadronic channel. In multilepton searches, studying the lepton rapidity distributions and charge assignments can be used to discriminate between $tuh$ couplings, for which anomalous single top production is relevant, and $tch$ couplings, for which it is suppressed by the parton distribution function of the charm quark. An analysis of fully hadronic $t+h$ production and $t\to h j$ decay can be competitive with the multilepton search at 100 fb$^{-1}$ of 13 TeV data if jet substructure techniques are employed to reconstruct boosted top quarks and Higgs bosons. To show this we develop a modified version of the HEPTopTagger algorithm, optimized for tagging $t \to h j$ decays. Our sensitivity estimates on $BR(t\to hu)$ ($BR(t\to hc)$) at 100 fb$^{-1}$ of 13 TeV data for multilepton searches, vector boson plus Higgs search and fully hadronic search are $0.22%$ ($0.33%$), $0.15%$ ($0.19%$) and $0.36%$ ($0.48%$), respectively.


Introduction
Determining the properties of the newly-discovered Higgs boson is one of the major goals of the LHC physics program. Higgs interactions with fermions are of special interest since deviations from Standard Model (SM) predictions could point to the existence of new flavor dynamics not too far above the electroweak scale. Among the flavor violating Higgs couplings to quarks, the most promising place to look for new physics at high energy colliders are processes involving top quarks. On the one hand, all relevant indirect low energy constraints on such processes are necessarily based on loop suppressed observables [1]. On the other hand, the large number of top quarks produced at the LHC allows us to study even strongly suppressed contributions to top quark production and decay. Using this feature, the CMS collaboration has provided the best official upper limit on flavor violating tch couplings: from a combination of multilepton searches and diphoton plus lepton searches, the constraint B(t → hc) < 0.56% is obtained at 95% confidence level (CL) [2].
In the present work, we explore the LHC sensitivity to non-standard flavor violating top-Higgs interactions (tch and tuh) further. Building upon related theoretical [3][4][5][6][7][8] and experimental [9][10][11] studies, we explore three main directions: (1) We demonstrate the importance of the single top+Higgs production processes in addition to t → hj decays. We demonstrate how these processes can be exploited to distinguish tch and tuh couplings in leptonic t + h events by studying lepton rapidity distributions and charge assignments.
(3) we consider several novel search signatures including hadronic top decays and Higgs decays to bb and τ + τ − . While this leads to more challenging signatures requiring efficient discrimination against the large SM backgrounds, the final sensitivity is compensated by increased signal yields. The remainder of the paper is organized as follows: In Sec. 2 we set up the notation and introduce our main physics ideas. Then we explore and quantify these insights in more detail using several top and Higgs decay modes. Multilepton searches [4] are particularly sensitive to (t → b ν) + (h → W + W − , ZZ, τ + τ − ) final states, and in Sec. 3.1 we recast a recent CMS analysis [9] to constrain these final states. In doing so, we demonstrate the importance of including the anomalous single top production process gu → th. In Sec. 3.2 we recast a recent CMS search [2] for flavor violating tch coupling in the diphoton plus lepton final state to set an improved bound on tuh coupling. In Sec. 3.3 we show that a competitive sensitivity can be obtained focusing specifically on h → τ + τ − decays by recasting a CMS search [12] for associate W + Higgs and Z + Higgs production. We then proceed to future searches, showing in Sec. 4.1 how a detailed analysis of kinematic distributions in multilepton searches can be used to improve the sensitivity to both tuh and tch couplings, and to discriminate between them. Finally, in Sec. 4.2, we develop a search strategy for the fully hadronic final state (t → bqq ) + (h → bb), where for highly boosted processes jet substructure techniques can be employed to identify top quarks and Higgs bosons. We summarize our results in Sec. 5.

Flavor Violating Top-Higgs Couplings
We parameterize the flavor violating top-Higgs interactions in the up-quark mass eigenbasis as − L tqh = y tutL u R h + y utūL t R h + y tctL c R h + y ctcL t R h + h.c. . (2.1) At tree level, this Lagrangian gives rise to the non-standard 3-body Higgs boson decays h → t * q → W bq as well as the more interesting 2-body top quark decays t → qh, where q = u, c (see Fig. 1). Neglecting the light quark masses and assuming the top quark decay g g ,q q th q ', th q ' u g th u g th c g th , c g th . Cross-sections for (t → bW ) + (t → hq) and single top + Higgs production induced by flavor violating top-Higgs couplings as a function of the hadronic center of mass energy and normalized to the corresponding tqh couplings. All partonic cross-sections are computed analytically at leading order in QCD, while parton luminosity integration is performed using MSTW2008 leading order parton distribution functions [17] with renormalization and factorization scales fixed to the top mass (µ r = µ f = m t = 173.2 GeV).
width is dominated by the SM value of Γ(t → W b), the approximate relation between the relevant t → qh branching ratios and the flavor violating Yukawa couplings is given by with the top quark mass m t , the W mass m W , the Higgs mass m h , and the Fermi constant G F . The above expression is based on the leading order formulae for both the t → W b and t → hq decay rates. The NLO QCD correction to the branching ratio (in the pole top mass scheme) are included through the factor η QCD = 1 + 0.97α s = 1.10, calculated using the known corrections to the t → W + b [13,14] and t → ch decay widths [15]. We note that values of y tq = y qt 0.13 correspond to B(t → hq) 1%. Top quark pair production followed by an anomalous t → qh decay has a total cross section of at the √ s = 8 (13) TeV energy LHC, where we have used the QCD NNLO values of σ(pp → tt) = 245 (806) pb [16].
The interactions in Eq. 2.1 also contribute to associated single top plus Higgs production at the LHC. In particular the effects of y tu and y ut are significant due to the large flux of valence u-quarks. The t + h production cross-section is comparable in magnitude to (2.  Figure 3. Pseudorapidity distributions for the Higgs boson in various flavor violating processes at 13 TeV for y tq = y qt = 0.13 (corresponding to B(t → hq) 1%) and an integrated luminosity of 1 fb −1 . The results are obtained using a FeynRules implementation of the effective interactions in Eq. (2.1) and using MadGraph for MC simulation. Events are normalized to corresponding state of the art QCD corrected cross sections as discussed in the Sec. 2.
where we have used the NLO QCD result of [5,18]. The cross section for the conjugate process antitop + Higgs production is roughly an order of magnitude smaller, and processes induced by tch couplings are even more suppressed as illustrated in Fig. 2. This implies that, for a given center of mass energy and luminosity, the sensitivity to tuh couplings is in general better than the one to tch couplings.
In addition, the presence or absence of a significant contribution of qg → th production in single top plus Higgs final states can be used to distinguish between couplings to up quarks and couplings to charm quarks. A good discriminating variable is the Higgs boson pseudorapidity, η h , as illustrated in Fig. 3. The relevance of this variable can be understood from the fact that in ug scattering, the interaction products tend to be boosted in the direction of the incoming valence u quark, which on average carries a larger fraction of the proton momentum than the gluon. In addition, the Higgs boson in such a scattering process is preferentially produced in the direction of the up quark in the partonic center of mass frame due to angular momentum conservation combined with the quark chirality flip at the tuh vertex. These effects add up to make the resulting η h distribution peak at large rapidities. For initial states not containing valence quarks (gluon fusion-induced tt production as well as single top + Higgs production in cg,cg, orūg collision), both the top quark and Higgs boson are produced more centrally. Another useful handle on tagging single top plus Higgs production in searches with leptonic top decays is the enhanced abundance of positively charged leptons.
3 Improved Limits on tuh and tch Couplings from Current LHC Searches

Recasting the CMS Multilepton Search
Multilepton searches at the LHC profit from relatively low SM backgrounds and are therefore sensitive to new physics processes producing final states with many leptons. A good example is a final state with a top quark and a Higgs boson [4], where the top quark decays to b ν, and the 126 GeV Higgs boson decays to final states with up to four leptons. The relevant processes are h → W W * → νν, h → τ τ , h → ZZ * → jj, h → ZZ * → νν, and h → ZZ * → with branching ratios 2.4%, 6.2%, 0.41%, 0.1% and 0.03%, respectively [23]. Single top + Higgs production can thus yield up to five leptons, so that multilepton searches can be expected to constrain anomalous flavor violating top-Higgs interactions.
In this section, we recast a recent CMS search for anomalous production of final states with three or more isolated leptons [9], based on 19.5 fb −1 of data at √ s = 8 TeV. Data are binned into exclusive categories according to the lepton flavor, the missing transverse energy E miss T , the scalar sum of the transverse momenta of all the jets H T , the existence of b-tagged jets, and the presence or absence of opposite sign, same flavor (OSSF) light lepton pairs. Events with an OSSF pair are further divided into "below Z", "on Z" and "above Z" categories based on the invariant mass of the OSSF lepton pair relative to the Z mass.
CMS has already interpreted this search as a constraint on the anomalous tch coupling [9], considering top pair production followed by anomalous top decay to h + j. However, the CMS search does not include contributions from single top + Higgs production, which is irrelevant for tch couplings, but very important for tuh couplings. Therefore, we study in the following the importance of associated th production for constraining anomalous tuh couplings.
We simulate the processes pp → tt followed by t → hu ort → hū decay, as well as pp → th and pp →th using MadGraph. We rescale the leading order cross sections to the corresponding higher order QCD results. In particular, pp → tt events are generated using the default MadGraph dynamical factorization and renormalization scales, and the final cross section is rescaled to σ(pp → tt) = 245 pb [16]. Single top plus Higgs events are generated using factorization and renormalization scales fixed to µ f = µ r = m h + m t , and a QCD correction factor of K QCD = 1.5 is applied [5]. Higgs bosons and gauge bosons are decayed using BRIDGE v2.24 [24], where the SM Higgs branching ratios are taken from [23]. Showering and hadronization are simulated in Pythia, and Delphes is used for detector simulation. We have modified the default implementation of the CMS detector in Delphes by switching to the anti-k T jet algorithm with distance parameter R = 0.5, by changing the light charged lepton isolation criteria in accordance with [9], and by implementing the b tagging efficiencies and mistag rates given in [9] for the medium working point of the Combined Secondary Vertex (CSV) algorithm. We apply analysis cuts in accordance with those used in the CMS multilepton search [9]. In particular, we require the leading charged lepton in each event to have p T > 20 GeV. Additional light charged leptons must have p T > 10 GeV, and all of them must be within |η| < 2.4. Events are rejected if they have an OSSF lepton pair with invariant mass m < 12 GeV. Jets are required to have |η| < 2.5 and p T > 30 GeV, and an angular distance ∆R > 0.3 from any isolated charged lepton candidates.
The results of our simulations are presented in Table 1. The most sensitive bins have exactly three isolated leptons and no hadronically decaying taus. Signal predictions are given for y ut = y tu = 0.13 which corresponds to B(t → hu) = 0.01. Taking into account the fact that we use a simplified detector simulation, the predictions for top pair production N (t → hj), are in good agreement with the results obtained by CMS [9]. This serves as an important cross check of our simulation. Table 1 confirms that for tuh couplings the contribution of associated th production to the signal, N (th), is of the same order as the contribution from tt production followed by t → hj decay, N (t → hj), as advocated before. Using the CL s method [25], we derive the new 95% CL limits The corresponding limits on the flavor violating couplings are |y tc | 2 + |y ct | 2 < 0.227 and |y tu | 2 + |y ut | 2 < 0.186. We have checked that the minor difference between Eq. (3.1) and the CMS result B(t → hc) < 1.28% is due to the contributions of hadronic tau decays which we do not include in our analysis. Our main conclusion, namely that the limit on B(t → uh) is more stringent than the limit on B(t → ch) by a factor of 1.5 due to associated th production, is unaffected by this omission.

Recasting the CMS Diphoton plus Lepton Search
Recently, CMS has interpreted a search for extended Higgs sectors in the diphoton plus lepton final state [26] as a constraint on flavor violating tch coupling [2], using 19.5 fb −1 of data collected at √ s = 8 TeV. In the following, we use this search to constrain also tuh couplings, taking into account the contribution from associated top plus Higgs production. We use MadGraph to simulate the signal processes induced by tuh couplings, namely, top pair production followed by anomalous t ort decay as well as associated single t (andt) plus Higgs production. Leptonic top decays as well as Higgs decays to pairs of photons are simulated using MadGraph where the implementation of the effective hγγ interaction is adopted from [27]. The SM branching ratio for h → γγ is taken to be 0.23% [23]. We rescale the leading order cross sections to the corresponding higher order QCD corrected results as in Sec. 3.1. We simulate showering and hadronization effects in Pythia and detector effects in Delphes. We use the same implementation of the CMS detector in Delphes as in Sec. 3.1.
We closely follow the CMS search [2] in our analysis. In particular, we require one light charged lepton with p T > 10 GeV and |η| < 2.4. We require two photons with p T > 40 GeV (p T > 25 GeV) for the leading (next to leading) photon and |η| < 2.5. The diphoton invariant mass is required to be between 120 and 130 GeV. Events are categorized into exclusive categories based on E miss T and on the presence or absence of a bottom-tagged jet.
We summarize the results of our simulations in Table 2. The most sensitive bins have a b-tagged jet and no hadronically decaying taus [2]. The predictions for signal yields are given for y ut = y tu = 0.13 which corresponds to B(t → hu) = 0.01. We validate our simulation by closely reproducing the predictions for top pair production followed by anomalous top decay, N (t → hj), presented in Table 3 of [2]. Finally, the contribution from associated th production, N (th), is competitive and thus important in the case of flavor violating tuh interactions. As before, we employ the CL s method [25] to derive the new 95% CL limits B(t → hc) < 0.66% and B(t → hu) < 0.45%, (3.3) where the corresponding limits on the flavor violating Yukawa couplings are |y tc | 2 + |y ct | 2 < 0.151 and |y tu | 2 + |y ut | 2 < 0.125. The obtained limit on tch couplings is in a good agreement with the CMS result |y tc | 2 + |y ct | 2 < 0.14 [2]. The search in the diphoton plus lepton final state sets the most competitive current bounds on flavor violating tqh interactions and will remain very promising for future studies. The current search is mainly limited by statistics, so that further improvements are expected at larger integrated luminosities. Improvements are also expected in the datadriven background estimation by fitting the background shapes from the sidebands around the Higgs mass window in the diphoton invariant mass [26]. We estimate the expected sensitivity to B(t → hq) at 100 fb −1 (3000 fb −1 ) and √ s = 13 TeV to improve by a factor ∼ 4 (∼ 25), based on naive scaling in cross section and luminosity. Our rough estimate is in a good agreement with the dedicated study performed by the ATLAS collaboration [28].
Furthermore, the advantage of this search with respect to other searches is an explicit reconstruction of the Higgs boson which would be very useful in the case of a positive signal. Finally, as we will show in Sec. 4.1, the origin of the signal (tuh or tch couplings) could be disentangled by studying the Higgs pseudorapidity distribution and the charges of the light charged lepton from the top decay.

Recasting the CMS Search for Vector Boson + Higgs Production
In [12], the CMS collaboration has searched for Higgs bosons produced in association with a W or Z and decaying to τ + τ − . This final state is very similar to the one obtained from single top + Higgs production, followed by t → W b and h → τ + τ − , and from tt production with one of the top quarks decaying to (h → τ + τ − ) + j. The CMS search can thus be recast to set limits on the flavor changing tuh and tch couplings that we are interested in here.
In doing so, we consider only the τ h final state consisting of two light leptons (electrons or muons) and one hadronically decaying τ . This final state turns out to be more sensitive than τ h τ h (one light lepton and two hadronic τ 's) in the CMS search, and is therefore also expected to give the best sensitivity in our case. In particular, the main competing factors affecting the relative importance of the τ h and τ h τ h channels-the small leptonic branching ratio of the τ and the larger fake rate for hadronic τ 's-affect the h + W, Z channel in the same way as our t + h final state. CMS also consider final states with four light charged leptons, with at least two of them consistent with a Z decay. Since in the case of t + h production or tt production followed by t → hj decay, only events with the suppressed Higgs decay h → ZZ * could contribute to this final state, we do not consider it here.
We simulate the t + h signal and the top and Higgs decays in MadGraph. Since in [12], CMS have used 5.0 fb −1 of data collected at √ s = 7 TeV as well as 19.5 fb −1 of data collected at √ s = 8 TeV, we simulate events for both center-of-mass energies. We rescale the leading order cross sections to the corresponding higher order QCD corrected results as in Sec. 3.1. We use TAUOLA v2.5 [29] to decay the τ leptons and Pythia for parton showering and hadronization. We choose Delphes as a detector simulation, and we adapt the default implementation of the CMS detector by adjusting the p T -dependent τ tagging   efficiency and mistag rate to the values given in [30] for the loose working point of the HPS ("hadron plus strips") algorithm.
In accordance with [12] we use the following cuts; we require exactly two light leptons (electrons or muons), with the p T of the leading lepton larger than 20 GeV and that of the subleading lepton larger than 10 GeV. Muons are required to have a pseudorapidity |η| < 2.4, while for electrons the requirement is |η| < 2.5. The leptons must have the same charge to suppress Z backgrounds, and the flavor combinations µµ and eµ are allowed while ee events are vetoed. We also require one τ -tagged jet with p T > 20 GeV and |η| < 2.3. Extra jets are allowed, but events containing a b-tagged jet with p T > 20 GeV and |η| < 2.4 are vetoed to suppress tt backgrounds. Finally, the scalar sum of the lepton and τ p T 's is required to be larger than 80 GeV.
To verify our simulation and our analysis, we have also simulated the Standard Model ZZ and W Z backgrounds. Fig. 4 shows that our background predictions are in excellent agreement with the CMS data [12] and with background predictions by CMS. The figure also shows that a t + h signal induced by flavor violating top-Higgs couplings at the current upper limit from CMS y 2 ut + y 2 tu = 0.14 would lead to a sizeable excess of events. Quantifying this excess using the CL s method [25], we find the new 95% CL limits on  Table 3. Limits on flavor changing tuh and tch couplings from recasting a CMS search for V +(h → τ τ ) production [12] into a search for anomalous t → jh decays and anomalous single top + Higgs production using 5.0 fb −1 of 7 TeV and 19.5 fb −1 of 8 TeV data. We also show the expected sensitivity of a similar search using 100 fb −1 of 13 TeV data.
flavor-violating top Yukawa couplings given in Table 3.
In the same table, we also give an estimate for the sensitivity of a future V + h search using 100 fb −1 of 13 TeV LHC data and assuming identical cuts as in the analysis at 7 and 8 TeV. Since we cannot reliably model the reducible background from fake leptons, we assume it to be of the same size and have the same m vis τ τ distribution as the W Z background. The larger instantaneous luminosity and larger pileup at 13 TeV may require somewhat harder cuts and could lead to increased backgrounds from misidentified jets. We expect, however, that these complications can be offset by further improvements of the analysis, for instance using multivariate techniques.

Future Multilepton Searches and Discrimination between tch and tuh Couplings
In this section, we study the potential of future multilepton searches at 13 TeV center of mass energy to constrain anomalous tqh interactions or to establish their existence. Furthermore, we study the ability to differentiate between tuh and tch couplings based on the presence or absence of large contributions from associated single top plus Higgs production to the signal. We closely follow the analysis conducted in Sec. 3.1. In particular, we use the same lepton and jet reconstruction and isolation requirements as before. An optimized search at 13 TeV will have slightly different requirements, such as somewhat higher lepton p T thresholds, but we expect these to have only a minor impact on the sensitivity. We require exactly three light charged leptons in the final state. In order to differentiate between tch and tuh signals, we bin the data further with respect to two variables: (1) the total sum of lepton charges Q tot , 1 and (2) the pseudorapidity η of the opposite charge dilepton system with the smallest angular distance ∆R ≡ ∆η 2 + ∆φ 2 . We expect a tuh signal to have a preference for Q tot = +1 due to a substantial contribution from the process ug → th, while tch couplings yield approximately equal numbers of events with Q tot = +1 and Q tot = −1. The idea behind the variable η is that the two leptons with the smallest ∆R have the highest probability of originating from h → W W * decay (as opposed to a semileptonic top decay), so that η is an approximation to the pseudorapidity of the Higgs boson in the event, which we have seen in Sec. 2 to be a promising discriminant between tuh and tch couplings. To illustrate the correlation between η and the Higgs rapidity η h , we have carried out a parton level simulation of the process ug → th followed by h → W W * → νν and t → W b → νb using MadGraph. In Fig. 5 we show the resulting distributions for η h , η and η h . The latter quantity is defined as the rapidity of the dilepton system that actually originates from Higgs decay. We see that, indeed, η nicely follows η h . Since we have already seen in Sec. 2 and Fig. 3 that η h is an efficient discriminator between tuh and tch couplings, we can expect the same to hold for the experimentally accessible quantity η . We use two bins in η : |η | > 1 and |η | < 1.
Recalling the results of the analysis from Sec. 3.1 based on real CMS data, we concentrate on the event categories that we have found to be most sensitive: we consider only events with exactly three light charged leptons that fall into the "above Z" , "no OSSF" or "below Z" categories; in the latter case we also require E miss T > 50 GeV. Moreover, we require at least one b-tagged jet. The dominant background in all categories is from fully leptonic tt events with a jet misidentified as a lepton [9]. We simulate pp → tt → + − ννbb at 8 TeV and 13 TeV center of mass energy using MadGraph and normalize the corresponding pp → tt cross sections to the NNLO QCD corrected values of σ(pp → tt) = 245 (806) pb [16], respectively. Showering, hadronization and detector effects are simulated using Pythia and Delphes. Following the procedure recommended by CMS [9], we model fake leptons by randomly converting an isolated track to a lepton with the measured conversion probability of 0.007 (0.006) for electron (muon) tracks. To check the validity of this approach, we first compare our 8 TeV predictions to CMS results [9] in  Table 4. Number of predicted signal and background events per bin for a multilepton analysis using 100 fb −1 of 13 TeV data. We have assumed B(t → hu) = 0.01. The column labeled N (t → hj) shows the signal contribution from t + (t → hq) events, while the column labeled N (th) contains the signal from single top plus Higgs production. The column labeled N (BG) is the expected background from SM tt events with a jet misidentified as a lepton. For flavor violating tch instead of tuh couplings, N (t → hj) remains unchanged, while N (th) becomes negligible because the process gc → th compared to gu → th by the small parton distribution function for charm quarks.
the dilepton control region that requires an opposite-sign eµ pair. We obtain good agreement with the H T and E miss T distributions shown in Fig. 1 and Fig. 2 of [9]. Second, we have checked that we agree with CMS, at the level of 30-40%, on the E miss T distributions (provided in [9]) of the tt background in the "noOSSF", "above Z" and "below Z" signal regions with low and high H T and with at least one b-tagged jet. The main difficulty in reproducing the background more precisely is the modeling of lepton misidentification.
Therefore, our quantitative results should be considered with care, and a dedicated experimental analysis is clearly necessary to obtain more precise predictions. We note in passing that the irreducible SM background coming from associate top + Higgs production with a cross-section of σ SM th 74 fb at 13 TeV LHC 2 is only expected to become relevant once the sensitivity reaches B(t → hq) ∼ 10 −4 .
Our predicted signal and background yields at 13 TeV center of mass energy are shown in Table 4 for an integrated luminosity of 100 fb −1 , and for B(t → hu) = 0.01. The most sensitive bins fall into the "below Z" categories. It is worth noting that single top + Higgs production (N (th) column in Table 4) tends to populate preferably bins with Q tot = +1 and |η | > 1, while the background (N (BG) column) and the t → qh signal (N (t → hj) column) are much more evenly distributed. This is of crucial importance in discriminating between the tuh and tch signal hypotheses.
To estimate the achievable sensitivity and discovery reach for flavor violating top-Higgs couplings, we use the CL s method [25], treating all bins as statistically independent Poisson variables. We treat the overall normalization of the background as a nuisance parameter (positively correlated among all bins) to account for the uncertainty in our modeling of the lepton misidentification probability. We do not impose any a priori constraints on the nuisance parameter, i.e. we determine it in the analysis together with the signal parameters, taking advantage of the fine-grained binning of the simulated data. Since in a realistic experimental analysis, the misidentification rate can be measured from a Z + jets control sample [9], our projected limits should be considered as very conservative.
The results of our statistical analysis are plotted in Fig. 6. Expected 95% CL limits on B(t → hu) [B(t → hc)] in the absence of a signal are shown as red (blue) thick solid curves. The expected 5σ discovery potential for a tuh (tch) signal is shown as a red (blue) thick dotted curve. The discrimination power between tuh and tch couplings is shown as thin dashed curves. For pure tuh (tch) couplings above the red (blue) thin dashed curve, the opposite hypothesis of pure tch (tuh) couplings can be ruled out at the 95% CL.
From the coincidence of the thick dotted curves and the thin dashed ones we conclude that, if a tqh signal is discovered (i.e. the BG only hypothesis is rejected at 5σ), the discrimination power between tuh and tch couplings is already at the level of 2σ. It is interesting that this remarkable performance is achieved in spite of the rather generic, unoptimized cuts in this multi-purpose multilepton analysis and of our rather conservative treatment of systematic uncertainties.

Searches in the Fully Hadronic Final State
The final state with the largest branching ratio in t + h production and t → hq decay is the fully hadronic one. Modern jet substructure techniques [33][34][35] offer promising tools to extract this signal from the otherwise overwhelming background of QCD multijet events, SM tt and single top production and vector boson plus jets production. They are efficient when the top quarks and Higgs bosons constituting the signal are highly boosted so that  For tuh (tch) couplings above the thin dashed curves, the tch (tuh) hypothesis can be excluded at 95% CL based on the different distributions of the dilepton rapidity η and the total charge Q tot . The discrimination power of these variables comes from the presence or absence of the process ug → th.
Since we treat the overall normalization of the background as an unconstrained nuisance parameter, our sensitivity projections are very conservative.
the angular separation of their decay products is too small to be resolved by conventional jet algorithms. Instead, jet substructure methods use "fat jets", i.e. jets with a very large radius R. After the initial clustering, the fat jet is partially unclustered again to examine the invariant mass of its largest subclusters. Comparing these invariant masses to the masses of possible parent particles such as top quarks, Higgs bosons or W bosons, the algorithm decides how probable it is that the fat jet was produced by one of these parent particles.
Here, we study the sensitivity of two analyses using jet substructure: (1) a search for tt events with one SM top decay and one flavor violating decay t → j + (h → bb); (2) a search for anomalous single top + Higgs production with SM top decay and h → bb.
In both analyses, we use the Cambridge-Aachen algorithm [36] as implemented in FastJet 3.0.3 [37] to cluster fat jets with a radius R = 1.5 and a minimum transverse momentum p T > 170 GeV. We run HEPTopTagger v1.0 [34,35] with default settings on these jets to identify those which are most likely to originate from a SM hadronic top decay t → b + (W → jj). HEPTopTagger imposes cuts on the invariant masses of the three main subjets of the top candidate, requiring that two of them reconstruct to a W , while all three together yield the top mass. Moreover, their combined p T has to exceed 200 GeV. In addition to these kinematic cuts, we also require the subjet that is most likely to originate from the b quark to contain a b tag (see Appendix A for details on our implementation of b-tagging).

Analysis 1: th tag + top tag
To identify flavor violating decays t → j + (h → bb) for analysis 1) and assign a "th" tag to the corresponding fat jets, we reprocess all fat jets using a modified version of HEPTop-Tagger, which we have optimized for this non-standard decay mode (see Appendix A). We require a b tag in each of the two subjets most likely to originate from the Higgs decay. We consider two different working points for our th tagger: a loose one with very robust kinematic cuts on the subjet invariant masses, and a tight one with somewhat more restrictive cuts that make it more efficient at suppressing backgrounds, but also more prone to systematic uncertainties in our simulations. Details on the kinematic cuts are given in Appendix A. A tight th tag moreover requires that the fat jet does not simultaneously carry a regular top tag.
Event selection for analysis 1 requires one fat jet with a loose or tight th tag and a second fat jet with a top tag.
We consider the backgrounds from tt production, single top production and QCD multijet production, but we have checked that W + jets, Z + jets, tt + h and SM single top + Higgs contributions are several orders of magnitude smaller than these dominant backgrounds. To simulate the tt and multijet backgrounds, we use Sherpa 1.4.3 [38][39][40][41][42] at leading order. For tt, we rescale the cross section to the NNLO value σ(pp → tt) = 806 pb [16], while QCD multijet events are rescaled by a factor K = 1.05, which has been empirically found to bring Sherpa predictions into agreement with data [43]. We note that in a realistic experimental analysis, backgrounds could be estimated directly from data. For tt and single top events, semileptonic final states offer a good control sample, while for QCD jet production, anti-b-tags can be employed to define a control region. For the simulation of the SM single top background and of the signal we use MadGraph, followed by Pythia for parton showering and hadronization.
The predicted event counts after cuts from analysis 1 are shown in the upper part of Table 5 for y 2 qt + y 2 tq = 0.1 and assuming 100 fb −1 of 13 TeV data. The predicted CL s sensitivity of the analysis is summarized in the upper part of Table 6. We see that an analysis of the fully hadronic final state can improve upon the current limits on flavor violating Higgs couplings, and that the future sensitivity is only slightly worse than the one expected from analyses involving leptons. A combined analysis of leptonic and hadronic final states would therefore seem worthwhile. Moreover, the hadronic channel would provide a crucial cross-check in case a signal is discovered in one of the other searches.

Analysis 2: Higgs tag + top tag
For analysis 2, we identify events in which a Higgs boson is directly produced ("Higgs tag") by using the mass drop tagger implemented in FastJet 3.0.3 [37,44]. Following [37], we require the two subjets obtained when the last step of clustering is undone to have jet masses at least a third smaller than the mass of the original fat jet. In addition, we require the asymmetry parameter y [37] Table 6. Projected sensitivity of searches for anomalous t → jh decays and anomalous single top + Higgs production in the fully hadronic final state, using 100 fb −1 of 13 TeV data. See text for a detailed explanation of the different analyses.
subjets have a sizeable angular separation and each of them carries a substantial fraction of the fat jet p T . If the latter is not the case for one of the subjets, it is discarded and the algorithm is restarted with the other subjet as input. To remove contamination from pileup and from the underlying event (which we do not explicitly simulate), we filter the fat jet by reclustering it with a smaller radius and keeping only the three hardest constituents (see Refs. [37,44] for details). We require b tags in the two hardest of them.
After top tagging and Higgs tagging, we preselect events by requiring that at least one fat jet in the event carries a Higgs tag and at least one of the remaining fat jets carries a top tag. We define the Higgs candidate as the hardest Higgs-tagged fat jet and the top candidate as the hardest top-tagged fat jet different from the Higgs candidate. If the hardest Higgs-tagged fat jet is the only fat jet carrying a top tag, we take it to be the top candidate and use the next-to-hardest Higgs-tagged fat jet as the Higgs candidate. Event counts after preselection are given in Table 5, and the distributions of two important kinematic quantities-the invariant mass m H and the pseudorapidity η h of the Higgs candidate-are shown in Fig. 7 for y 2 ut + y 2 tu = 0.1. As expected, m H peaks around the true value of the Higgs mass for the signal, while showing no distinct features for the background. The forward bias of the η h distribution for signal events is again related to angular momentum conservation in the center of mass frame and the net boost of that frame in the direction of the incoming up quark in the process gu → th (see Sec. 2 for details), making η h again a good discriminant between tuh and tch couplings. The m H and η h distributions shown in Fig. 7 suggest the final cuts 100 GeV < m H < 130 GeV and |η h | > 1.5. We see from the predicted event counts in the last row of Table 5 that the signal-to-background ratio S/B is substantially improved by these cuts. Even though the signal-to-square root background ratio S/ √ B is similar before and after the final cuts, this improvement makes the search much more robust with respect to systematic uncertainties.
From the event counts in Table 5 and the projected sensitivities in Table 6, we see that analysis 2 outperforms analysis 1 in the case of tuh couplings, but is not competitive for tch couplings, as expected. It could therefore be an important ingredient in a multichannel search for tuh couplings, and an important cross check in case a signal is found in a different channel.

Discussion and Conclusions
In this work we have investigated the sensitivity of the LHC to flavor violating top-Higgs interactions. Since these interactions are highly suppressed in the SM, a positive signal at the LHC would constitute a clear sign of new physics, for instance in the form of additional Higgs bosons or nonrenormalizable couplings of the Higgs.
While exiting experimental searches have mainly concentrated on anomalous top decays t → hq, we have shown that anomalous single top plus Higgs production is almost as important in the case of tuh couplings and therefore offers a promising avenue for further improvements in the sensitivity. Single top + Higgs production is less relevant for probing tch interactions due to the suppressed charm quark parton distribution in the proton.
In including the contribution form single top + Higgs production. Our best limits on the branching ratio B(t → hu) < 0.45% and the Yukawa couplings y 2 ut + y 2 tu < 0.014 come from the diphoton plus leptons final state and are a factor 1.5 stronger than the previously derived limits on B(t → hc) and y 2 ct + y 2 tc . Limits from multileptons and vector boson + Higgs searches are slightly weaker, but still competitive. Our new limits are summarized in the upper part of Table 7.
In the second part of the paper, Sec. 4, we have investigated possible future improvements of searches for flavor violating top-Higgs couplings, including the development of a completely new search strategy in fully hadronic final states. We have shown that multilepton, diphoton + lepton and vector boson + Higgs searches can substantially improve the current bounds and may have the potential to distinguish tuh couplings from tch couplings at the 2σ level once a signal is discovered at 5σ. This is possible because, in the case of tuh couplings, the process ug → th contributes significantly to the signal. In this process, the Higgs boson tends to be produced with a large forward boost, while in all other signal processes the Higgs rapidity distribution is more central. Moreover, ug → th leads to an asymmetry of the total charge of the final state leptons. For tch couplings, the corresponding process cg → th is suppressed by the parton distribution function of the charm quark and is therefore negligible.
Regarding the fully hadronic processes (t → bjj) + (h → bb) and t → j + (h → bb), we have developed an analysis using jet substructure techniques to tag SM top decays, h → bb decays and t → j + (h → bb) decays. We find that backgrounds can be suppressed efficiently in such a search, leading to a sensitivity that is competitive to that of searches with leptonic or semileptonic final states. Our projected future limits are summarized in the lower part of Table 7.
For completeness we note that several other LHC processes exhibit potential sensitivity to flavor violating top-Higgs interactions. For example, tuh couplings can lead to an enhancement of di-Higgs production at tree level through u-u collisions with t-channel top exchange. However, in this case the relevant cross-section scales with the fourth power of the flavor violating Yukawa couplings, and at the current upper limit the resulting effect is already subleading compared to the (already very suppressed) SM rate [45]. 3 Similarly, same-sign top production from u-u scattering via Higgs exchange in the t-channel is expected to be below the current experimental sensitivity (cf. [48]) given currently allowed values of y tu,ut .
To summarize, several signatures of flavor violating tqh interactions at the LHC which we have studied in the present paper exhibit comparable prospects to constrain or discover such phenomena. Moreover, it may be possible to even discriminate between tuh and tch signals by exploiting the presence of absence of the partonic process ug → th. When multiple searches are combined into a global analysis, they could allow the LHC experiments to probe the flavor violating top-Higgs interactions well into the region of B(t → hj) 0.1%. The motivation for these cuts can be seen in Fig. 8, which shows the distributions of m 23 /m 123 and arctan(m 13 /m 12 ) for signal and background events. The conditions on the left in Eq. (A.1) loosely define the physically accessible region, while the cuts on the right impose the condition that one of the subjet pairs reconstructs to the mass of an on-shell intermediate particle: the W for the original HEPTopTagger and the Higgs for our t → h tagger. Based on Fig. 8, we choose R min = 0.9 m H /m t , R max = 1.1 m H /m t in our most conservative analysis (loose th tags), and R min = 0.9 m H /m t , R max = 1.0 m H /m t in our more optimistic analysis (tight th tags). 4 The latter is based on the observation that the invariant masses of the subjets from Higgs decay tend to be slightly smaller than the true m H on average. We attribute this to individual hadrons falling outside the fat jet cone, being reconstructed as part of the wrong subjet, or being removed by filtering. Note that these effects are largest for the softest subjets. As discussed in Sec. 4.2, we require the two jets most likely originating from a Higgs decay to contain b tags. In the absence of a full detector simulation we perform b tagging by searching for b or c quarks with p T > 25 GeV within an angular distance ∆R < 0.4 from the reconstructed subjet axis. If a b or c quark satisfying these requirements exists inside the subjet, we assign a b tag with a probability depending on the quark's transverse momentum p T and its pseudorapidity η according to for b quarks: If no sufficiently hard and central b or c quark is found, the probability that the jet is still misidentified as a b jet is (u,d,s) b = 0.001. In practice, we do not actually discard events, but merely reweight them with the appropriate tagging efficiency.