1 Introduction

If new particles beyond the standard model (SM) are much heavier than 100 GeV, physics at the weak scale can be described by an effective field theory (EFT) with the SM Lagrangian perturbed by higher-dimensional operators. The latter encode, in a model-independent way, possible effects of new heavy particles at energies well below the new physics scale \(\Lambda \). The EFT framework allows for a systematic expansion of these effects in operator dimensions or, equivalently, in powers of \(1/\Lambda \). The leading effects are expected from operators of dimension 6, as their coefficients are suppressed by \(1/\Lambda ^2\). The first classification of dimension-6 operators was performed in Ref. [1]. For one generation of fermions, a complete non-redundant set (henceforth referred to as a basis) was identified and explicitly written down in Ref. [2]. Reference [3] extended this to three generations of fermions, in which case a basis is characterized by 2499 independent parameters.

In this paper we are interested in the subset of these operators that lead to exotic decays of the 125 GeV Higgs boson. By “exotic” we mean decays that are forbidden in the SM or predicted to occur with an extremely suppressed branching fraction. More specifically, we are interested in decays that violate the lepton flavor or quark flavor. Lepton-flavor-violating (LFV) processes are completely forbidden in the SM in the limit of zero neutrino masses. Quark-flavor-violating (QFV) Higgs decays as a flavor-changing neutral current process are forbidden in the SM at tree level. Theoretical studies of exotic Higgs decays have a long history; see e.g. Refs. [4–11] and [12] for a review. Most of these papers assume new light degrees of freedom, in which case the EFT approach described here is not adequate. On the other hand, Refs. [13, 14] recently studied the possibility of LFV and QFV two-body decays of the 125 GeV Higgs within the EFT framework. Such decays can arise in the presence of Yukawa-type dimension-6 operators [15]. These papers demonstrated that LFV Higgs decays to \(\tau ^\pm \mu ^\mp \) and \(\tau ^\pm e^\mp \) with the branching fraction as large as 10 % are allowed by current indirect constraints. At the same time, the LHC is currently sensitive to branching fractions of order 1 % [16]. This corresponds to probing the scale suppressing the corresponding dimension-6 operators at the level of \(\Lambda \sim 10\) TeV.

The goal for this paper is to extend this study to a full set of dimension-6 operators. Apart from the Yukawa-type operators, exotic Higgs decays can arise in the presence of vertex-type \(\sim H^\dagger D_\mu H \bar{\psi }\gamma ^\mu \psi \) and dipole-type \(\sim H \bar{\psi }\sigma ^{\mu \nu } \psi F_{\mu \nu }\) operators. We systematically discuss these operators and the new Higgs decay channels that they imply. The structure of the dimension-6 Lagrangian then implies certain relations between these Higgs couplings, as well as relations between single-Higgs interactions and Lagrangian terms without a Higgs that affect precision observables. We give the limits on each of the couplings from precision tests of the SM. That information can be explored to place limits on the allowed magnitude of the Higgs couplings. We will discuss the maximum exotic Higgs branching fraction that these limits permit.

The paper is organized as follows. In Sect. 2 we define our notation and introduce the dimension-6 Lagrangian with LFV and QFV interactions in the Higgs basis. In Sect. 3 we review and update the results of Refs. [13, 14] concerning two-body exotic Higgs decays. In Sect. 4 we study the possibility of LFV and QFV Higgs decays mediated by vertex- and dipole-type operators, respectively. Obviously, studying the full parameter space of the dimension-6 Lagrangian would be an extremely difficult task. To deal with the degeneracies among the parameters, one simplifying assumption we make throughout this paper is that the flavor-diagonal Higgs couplings are not significantly affected by higher-dimensional operators. Furthermore, we will assume that there are no large fine-tuned cancellations between different parameters so as to satisfy constraints from precision experiments. In such a constrained framework, we discuss the limits on the LFV and QFV Higgs couplings from various precision measurements. Given these constraints, we discuss the implications for the rate of exotic Higgs decays at the LHC.

2 Exotic Higgs couplings from dimension-6 Lagrangian

We consider an effective theory where the SM is extended by dimension-6 operators:

$$\begin{aligned} \mathcal{L}_\mathrm{eff} = \mathcal{L}^\mathrm{SM} + \frac{1}{v^2} \mathcal{L}^{D=6}. \end{aligned}$$
(1)

We assume the SM electroweak symmetry is linearly realized. This implies \(\mathcal{L}_\mathrm{eff}\) contains local operators invariant under the \(SU(3) \times SU(2) \times U(1)\) symmetry; in particular, the Higgs boson h enters the Lagrangian only through gauge invariant interactions of the Higgs doublet H. The SM Lagrangian in our notation takes the form

$$\begin{aligned} \begin{array}{ll} \mathcal{L}^\mathrm{SM} =\;&{} -\frac{1}{4} G^a_{\mu \nu } G^{a\;{\mu \nu }} - \frac{1}{4} W^i_{\mu \nu } W^{i\;{\mu \nu }} - \frac{1}{4} B_{\mu \nu } B^{\mu \nu }\\ &{}+ D^\mu H^\dagger D_\mu H + \mu _H^2 H^\dagger H - \lambda (H^\dagger H)^2 \\ &{} + \displaystyle \sum _{f \in q, \ell } i \bar{f}_L \gamma ^\mu D_\mu f_L + \sum _{f \in u ,d,e} i \bar{f}_R \gamma ^\mu D_\mu f_R \\ &{} - \left[ \tilde{H}^\dagger \bar{u}_R Y^u q_L + H^\dagger \bar{d}_R Y^d V_\mathrm{CKM} q_L + H^\dagger \bar{e}_R Y^\ell \ell _L + \mathrm {h.c.}\right] . \end{array} \end{aligned}$$
(2)

The gauge couplings of \(SU(3) \times SU(2) \times U(1)\) are denoted by \(g_S\), \(g_L\), \(g_Y\), respectively; we also define the electromagnetic coupling \(e = g_L g_Y/\sqrt{g_L^2 + g_Y^2}\), and the weak angle \(s_\theta = g_Y/\sqrt{g_L^2 + g_Y^2}\). The Higgs doublet H acquires the VEV \(\langle H \rangle = (0,v/\sqrt{2})\), where \(v \approx 246.2\) GeV. We also define \(\tilde{H}_i = \epsilon _{ij} H_j^*\). After electroweak symmetry breaking, the gauge mass eigenstates are defined as \(W^\pm = (W^1 \mp i W^2)/\sqrt{2}\), \(Z = c_\theta W^3 - s_\theta B\), \(A = s_\theta W^3 + c_\theta B\), where \(c_\theta = \sqrt{1 - s_\theta ^2} = g_L/\sqrt{g_L^2 + g_Y^2}\). The fermions \(q_L = ( u_L , V_\mathrm{CKM}^\dagger d_L)\) and \(\ell _L = (\nu _L, e_L)\) are doublets of the SU(2) gauge group. All fermions are three-component vectors in the generation space. We work in the basis where the fermions are mass eigenstates, thus \(Y^{u,d,\ell }\) are \(3 \times 3\) diagonal matrices such that \([Y^f]_{ij} \frac{v}{\sqrt{2}} = m_{f_i} \delta _{ij}\). The Higgs boson interactions following from Eq. (2),

$$\begin{aligned} \mathcal{L}^\mathrm{SM}_h= & {} \left( \frac{h}{v} + \frac{h^2}{2 v^2} \right) \left[ 2 m_W^2 W^+_\mu W^{-\;\mu } + m_Z^2 Z_\mu Z^\mu \right] \nonumber \\&- \frac{h}{v} \displaystyle \sum _f m_f \bar{f} f - \frac{m_h^2}{2 v} h^3 - \frac{m_h^2}{8 v^2} h^4 , \end{aligned}$$
(3)

do not contain any LFV nor QFV couplings.

We move to describe the effect of dimension-6 operators. In Eq. (1) we choose to normalize them by the electroweak scale v, while the new physics scale \(\Lambda \) is absorbed into the coefficients \(c_i \sim v^2/\Lambda ^2\) of these operators in the Lagrangian. A complete non-redundant \(\mathcal{L}^{D=6}\) for three generations of fermions was explicitly written down in Ref. [3]. Here we work at the level of Higgs boson couplings with other SM mass eigenstates, as in [17, 18]. In this language, the Lagrangian is defined by a set of couplings \([\delta y_f]\), \([\delta g^{Vf}]\), and \([d_{Vf}]\), which are in general \(3\times 3\) matrices with non-diagonal elements for all fermion species f. A subset of these interactions violates lepton flavor and introduces tree-level flavor changing neutral currents for quark-flavor violation.

The first group is related to corrections to the SM Higgs Yukawa couplings in Eq. (3):

$$\begin{aligned} \mathcal{L}_{hff}^{D=6} = - \frac{h}{v} \displaystyle \sum _{f \in u,d,e} \sum _{i \ne j} \sqrt{m_{f_i} m_{f_j}} \left[ [\delta y_f]_{ij} \bar{f}_{R,i} f_{L,j} + \mathrm {h.c.}\right] . \end{aligned}$$
(4)

These couplings arise from dimension-6 operators of the form \(c_f |H|^2 \bar{f} H f\), with \([c_f]_{ij} \sim \sqrt{m_{f_i} m_{f_j}} [\delta y_f]_{ij}\).

The second group is related to the contact interactions between the Higgs boson, fermions, and the massive SU(2) vector bosons:

$$\begin{aligned} \mathcal{L}_{hVff}^{D=6}= & {} \frac{g_L}{\sqrt{2}} \left( 1 + \frac{h}{v} \right) ^2 W_\mu ^+ \displaystyle \sum _{i \ne j} \left( \bar{u}_{L,i} \gamma ^\mu [\delta g^{Wq}_L]_{ij} d_{L,j} \right. \nonumber \\&\left. +\, \bar{u}_{R,i} \gamma ^\mu [\delta g^{Wq}_R]_{ij} d_{R,j} + \bar{\nu }_{L,i} \gamma ^\mu [\delta g^{W\ell }_L]_{ij} e_{L,j} \right) + \mathrm {h.c.}\nonumber \\&+ \sqrt{g_L^2 + g_Y^2} \left( 1 + \frac{h}{v} \right) ^2 Z_\mu \displaystyle \sum _{ij} \left[ \displaystyle \sum _{f \in u,d,e,\nu } \bar{f}_{L,i} \gamma ^\mu \right. \nonumber \\&\left. [\delta g^{Zf}_L]_{ij} f_{L,j} + \displaystyle \sum _{f \in u,d,e} \bar{f}_{R,i} \gamma ^\mu [\delta g^{Zf}_R]_{ij} f_{R,j} \right] \end{aligned}$$
(5)

where \([\delta g^{Vf}]\) are Hermitian matrices. These couplings arise from dimension-6 operators of the form \(H^\dagger D_\mu H \bar{f} \gamma ^\mu f\). The gauge symmetry of the dimension-6 Lagrangian implies \(\delta g^{Wq}_L = \delta g^{Zu}_L V_\mathrm{CKM} - V_\mathrm{CKM} \delta g^{Zd}_L\) and \(\delta g^{W\ell }_L = \delta g^{Z\nu }_L - \delta g^{Ze}_L\). Furthermore, it implies that the Higgs boson enters via \((1 + h/v)^2\). Therefore, the strength of the Higgs contact interactions of this form is correlated with vertex corrections to the W and Z boson interactions with fermions.

Finally, we also consider the dipole-type Higgs interactions:

$$\begin{aligned} \mathcal{L}_\mathrm{dipole}^{D=6}= & {} - \frac{1 + h/v}{v^2} \nonumber \\&\times \sum _{i \ne j} \left[ g_S \sum _{f \in u,d} \sqrt{m_{f_i} m_{f_j}} \bar{f}_{R,i} \sigma ^{\mu \nu } T^a [d_{Gf}]_{ij} f_{L,j} G_{\mu \nu }^a\right. \nonumber \\&+\, e \displaystyle \sum _{f \in u,d,e} \sqrt{m_{f_i} m_{f_j}} \bar{f}_{R,i} \sigma ^{\mu \nu } [d_{A f}]_{ij} f_{L,j} A_{\mu \nu } \nonumber \\&+ \sqrt{g_L^2 + g_Y^2} \displaystyle \sum _{f \in u,d,e} \sqrt{m_{f_i} m_{f_j}} \bar{f}_{R,i} \sigma ^{\mu \nu } [d_{Zf}]_{ij} f_{L,j} Z_{\mu \nu }\nonumber \\&+ \sqrt{2} g_L \left( \sqrt{m_{u_i} m_{u_j}} \bar{u}_{R,i} \sigma ^{\mu \nu } [d_{Wu}]_{ij} d_{L,j} W_{\mu \nu }^+\right. \nonumber \\&\left. + \sqrt{m_{d_i} m_{d_j}} \bar{d}_{R,i} \sigma ^{\mu \nu } [d_{Wd}]_{ij} u_{L,j} W_{\mu \nu }^- \right) \nonumber \\&\left. + \sqrt{2} g_L \left( \sqrt{m_{e_i} m_{e_j}} \bar{\nu }_{L,i} \sigma ^{\mu \nu } [d_{We}]_{ij} e_{R,j} W_{\mu \nu }^+ \right) \right] +\mathrm {h.c.},\nonumber \\ \end{aligned}$$
(6)

where \(\sigma _{\mu \nu } = \frac{\imath }{2}[\gamma _\mu , \gamma _\nu ]\), and \([d_{Vf}]\) are general \(3\, \times \, 3\) matrices. These couplings are absent in the SM at the tree level, but they arise from dimension-6 operators of the form \(H \bar{f} \sigma ^{\mu \nu } f V_{\mu \nu }\). The gauge symmetry of the dimension-6 Lagrangian implies that the W boson dipole couplings are related to those of the Z boson and the photon: \(\eta _f d_{Wf} = d_{Zf} + s_\theta ^2 d_{Af}\), \(\eta _u = 1\), \(\eta _{d,e} = -1\). Again, it also dictates that the Higgs boson enters via \((1 + h/v)\). Therefore, the strength of this type of Higgs interactions is correlated with the strength of dipole interactions of the SM fermions and gauge bosons.

In Eqs. (4) and (6) we isolated the factor \(\sqrt{m_{f_i} m_{f_j}}\) in the Yukawa and dipole interactions. This is done for convenience, and we do not assume any particular pattern of \([\delta y_f]_{ij}\) and \([d_{Vf}]_{ij}\). The Yukawa and dipole interactions are distinguished by the fact that they violate chirality (they allow for transitions of left-handed fermions into right-handed ones and vice versa), much like the fermion mass terms in the SM. Any model addressing the flavor problem and generating these parameters in the low-energy EFT is expected to exhibit some sort of chiral suppression. Exactly this pattern will arise from models following the minimal flavor-violation paradigm, where all sources of flavor violation are proportional to the SM Yukawa matrices. Although, more generally, the chiral suppression does not have to be proportional to the fermion masses, isolating the mass factor leads to a more transparent picture for natural values of these parameters. For the Yukawa interactions, the off-diagonal couplings can be more readily compared to the diagonal ones which, in this normalization, are just equal to 1 in the SM limit.

In the rest of this paper, we discuss LFV and QFV exotic Higgs decays induced by the operators in Eqs. (4), (5), and (6). As mentioned before, we assume that the flavor-diagonal Higgs couplings are not significantly affected by higher-dimensional operators,Footnote 1 and that there are no large fine-tuned cancellations between different parameters so as to satisfy constraints from precision experiments. In such a constrained framework, we discuss the limits on the LFV and QFV Higgs couplings from various precision measurements. With these assumptions, we give the limits on the couplings from precision experiments and discuss the maximum exotic Higgs branching fractions allowed.

3 Two-body Higgs decays

In this section we discuss two-body flavor-violating decays involving the Higgs boson. Such processes are generated via the Yukawa couplings in Eq. (4). The important point is that the \([\delta y_f]_{ij}\) are free parameters from the EFT point of view, and can take any value within the EFT validity range.

3.1 Lepton-flavor-violating decays

No experimental dedicated searches have been done so far for \(h \rightarrow \mu e\) and \(h \rightarrow \tau e\). For \(h \rightarrow \tau \mu \), the 95 % CL upper limit on the branching ratio was set by CMS [20] and ATLAS [21]:

$$\begin{aligned} \mathrm{Br}(h \rightarrow \tau \mu )\le & {} 1.51~\% \quad (\mathrm{CMS}); \nonumber \\ \mathrm{Br}(h \rightarrow \tau \mu )\le & {} 1.85~\% \quad (\mathrm{ATLAS}). \end{aligned}$$
(7)

The CMS search shows a \(2.4\sigma \) excess over the expected null background,Footnote 2 \(\mathrm{Br}(h \rightarrow \tau \mu ) = \left( 0.84^{+0.39}_{-0.37} \right) \%\), while the “excess” in ATLAS is only \(1\sigma \), \(\mathrm{Br}(h \rightarrow \tau \mu ) = \left( 0.77^{+0.62}_{-0.62} \right) \%\). A naive combination of the ATLAS and CMS results yields

$$\begin{aligned} \mathrm{Br}(h \rightarrow \tau \mu )= & {} \left( 0.82^{+0.33}_{-0.32} \right) ~\% ;\nonumber \\ \mathrm{Br}(h \rightarrow \tau \mu )\le & {} 1.47~\% \quad (\mathrm{ATLAS+CMS}). \end{aligned}$$
(8)

In terms of the parameters in Eq. (4), the branching ratio can be written as

$$\begin{aligned} \frac{\mathrm{Br}(h \rightarrow \tau \mu )}{\mathrm{Br}(h \rightarrow \tau \tau )} = \frac{m_\mu }{m_\tau } \left( | [\delta y_\ell ]_{\mu \tau } |^2 + | [\delta y_\ell ]_{\tau \mu } |^2 \right) , \end{aligned}$$
(9)

where we assumed the \(h \rightarrow \tau \tau \) decay is not significantly affected by new physics. Using \(m_\mu = {105.7}~\mathrm{MeV}\), \(m_\tau = {1.78}~\mathrm{GeV}\), \(\mathrm{Br}(h \rightarrow \tau \tau ) = 6.3~\%\) from the SM value, we obtain the best fit value and the 95 % CL bound on the EFT parameters:

$$\begin{aligned} \begin{array}{lll} \displaystyle | [\delta y_\ell ]_{\mu \tau } |^2 + | [\delta y_\ell ]_{\tau \mu } |^2 = 2.19^{+0.88}_{-0.85}, \\ \displaystyle \sqrt{| [\delta y_\ell ]_{\mu \tau } |^2 + | [\delta y_\ell ]_{\tau \mu } |^2} \le 1.98. \end{array} \end{aligned}$$
(10)

The strongest constraints on the LFV Higgs couplings come from \(\ell _2 \rightarrow \ell _1 \gamma \) decays [13, 14]. In the SM, such processes are completely forbidden in the limit of zero neutrino masses, but they can be generated in the presence of D=6 operators. In the EFT with LFV Yukawa couplings, they occur at one-loop level. The amplitude for the process is parametrized as

$$\begin{aligned}&\mathcal {M} = \overline{u(\ell _1)} F_2 \sigma ^{\mu \nu } k_\nu u(\ell _2) \epsilon ^\star _\mu (k) \quad \text {with:}\nonumber \\&F_2 = \frac{1}{16\pi ^2} (C_L P_L + C_R P_R) ; \quad P_{L/R} = \frac{1 \mp \gamma _5}{2}, \end{aligned}$$
(11)

and the decay width is given by (in the approximation \(m_{\ell _1} \ll m_{\ell _2}\)):

$$\begin{aligned} \Gamma _{\ell _2 \rightarrow \ell _1 \gamma } \approx \frac{m_{\ell _2}^3}{4096 \pi ^5} \left( |C_L|^2 + |C_R|^2 \right) . \end{aligned}$$
(12)

Evaluating the one-loop diagrams we find the following results:

  • \(\underline{\mu \rightarrow e \gamma {:}}\)

    $$\begin{aligned} \begin{pmatrix} C_L \\ C_R \end{pmatrix}\approx & {} e \frac{m_\tau ^2 \sqrt{m_\mu m_e}}{2 m_H^2 v^2} \left( 2 \ln \left( \frac{m_H^2}{m_\tau ^2}\right) - 3\right) \nonumber \\&\times \begin{pmatrix} [\delta y_\ell ]_{e \tau } [\delta y_\ell ]_{\tau \mu } \\ [\delta y_\ell ]_{\mu \tau }^* [\delta y_\ell ]_{\tau e}^* \end{pmatrix} \nonumber \\\approx & {} 5.4 \times 10^{-11}\, \mathrm{GeV}^{-1} \begin{pmatrix} [\delta y_\ell ]_{e \tau } [\delta y_\ell ]_{\tau \mu } \\ [\delta y_\ell ]_{\mu \tau }^* [\delta y_\ell ]_{\tau e}^* \end{pmatrix} , \end{aligned}$$
    (13)

  • \(\underline{\tau \rightarrow e \gamma {:}}\)

    $$\begin{aligned} \begin{pmatrix} C_L \\ C_R \end{pmatrix}&\approx e \frac{m_\tau ^2 \sqrt{m_e m_\tau }}{3 m_H^2 v^2} \left( 3 \ln \left( \frac{m_H^2}{m_\tau ^2}\right) - 4\right) \begin{pmatrix} [\delta y_\ell ]_{e \tau } \\ [\delta y_\ell ]_{\tau e}^* \end{pmatrix}\nonumber \\&\approx 2.2 \times 10^{-10}\, \mathrm{GeV}^{-1} \begin{pmatrix} [\delta y_\ell ]_{e \tau } \\ [\delta y_\ell ]_{\tau e}^* \end{pmatrix} , \end{aligned}$$
    (14)

  • \(\underline{\tau \rightarrow \mu \gamma {:}}\)

    $$\begin{aligned} \begin{pmatrix} C_L \\ C_R \end{pmatrix}&\approx e \frac{m_\tau ^2 \sqrt{m_\mu m_\tau }}{3 m_H^2 v^2} \left( 3 \ln \left( \frac{m_H^2}{m_\tau ^2}\right) - 4\right) \begin{pmatrix} [\delta y_\ell ]_{\mu \tau } \\ [\delta y_\ell ]_{\tau \mu }^* \end{pmatrix}\nonumber \\&\approx 3.2 \times 10^{-9}\, \mathrm{GeV}^{-1} \begin{pmatrix} [\delta y_\ell ]_{\mu \tau } \\ [\delta y_\ell ]_{\tau \mu }^* \end{pmatrix} . \end{aligned}$$
    (15)

Above, we kept only the contributions from diagrams with the \(\tau \) lepton in the internal fermion line. Other contributions are suppressed by \(m_\mu /m_\tau \) or \(m_e/m_\tau \) and can be neglected, unless there is a huge hierarchy between different off-diagonal elements of \([\delta y_f]\). Such hierarchy is not expected for EFT arising as low-energy approximation of specific models where the flavor problem is addressed. Our results agree with Refs. [13, 14].

It was pointed out in the literature [13, 14, 23, 24] that certain two-loop corrections, the so-called Barr–Zee diagrams with a W or a top loop, may give comparable contributions as the one-loop diagrams computed above. Their analytical forms can be found in Appendix A.2 of [14], which were adapted from the \(\mu \rightarrow e \gamma \) formulas of Chang et al. [25] and Leigh et al. [26]. It turns out that Barr–Zee contributions are proportional to \(\sqrt{m_i m_j} \delta y_{ij} C\), where C is common for all the processes. Numerically, one finds

  • \(\underline{\mu \rightarrow e \gamma {:}}\)

    $$\begin{aligned} \begin{pmatrix} C_L \\ C_R \end{pmatrix} \approx 2.3 \times 10^{-10} \,\mathrm{GeV}^{-1} \begin{pmatrix} [\delta y_\ell ]_{e \mu } \\ [\delta y_\ell ]_{\mu e}^* \end{pmatrix} . \end{aligned}$$
    (16)

  • \(\underline{\tau \rightarrow e \gamma {:}}\)

    $$\begin{aligned} \begin{pmatrix} C_L \\ C_R \end{pmatrix} \approx 9.6 \times 10^{-10}\, \mathrm{GeV}^{-1} \begin{pmatrix} [\delta y_\ell ]_{e \tau } \\ [\delta y_\ell ]_{\tau e}^* \end{pmatrix} . \end{aligned}$$
    (17)

  • \(\underline{\tau \rightarrow \mu \gamma {:}}\)

    $$\begin{aligned} \begin{pmatrix} C_L \\ C_R \end{pmatrix} \approx 1.4 \times 10^{-8}\, \mathrm{GeV}^{-1} \begin{pmatrix} [\delta y_\ell ]_{\mu \tau } \\ [\delta y_\ell ]_{\tau \mu }^* \end{pmatrix} . \end{aligned}$$
    (18)

Indeed, the two-loop contributions turn out to be dominant, for \(\tau \rightarrow \mu \gamma \) and \(\tau \rightarrow e \gamma \) by approximately a factor of 4. For \(\mu \rightarrow e \gamma \) the ratio of two- and one-loop contributions depends on the ratios of the different off-diagonal Yukawa couplings.

Table 1 Experimental 90 % CL upper limits on the branching fraction for lepton radiative flavor-violating processes
Full size table

The experimental limits on these processes obtained by the BaBar collaboration (\(\tau \rightarrow \ell \gamma \)), and the MEG experiment (\(\mu \rightarrow e \gamma \)) are collected in Table 1. Using those, we find the following constraints on the lepton-flavor-violating Yukawa couplings:

  • \(\underline{\mu \rightarrow e \gamma {:}}\)

    $$\begin{aligned}&\sqrt{ \left| [\delta y_\ell ]_{ e \mu } + 0.2 [\delta y_\ell ]_{e \tau } [\delta y_\ell ]_{\tau \mu } \right| ^2 + \left| [\delta y_\ell ]_{ \mu e} + 0.2 [\delta y_\ell ]_{\mu \tau } [\delta y_\ell ]_{\tau e} \right| ^2 }\nonumber \\&\quad \le 0.048. \end{aligned}$$
    (19)

  • \(\underline{\tau \rightarrow e \gamma {:}}\)

    $$\begin{aligned} \sqrt{ \left| [\delta y_\ell ]_{ e \tau } \right| ^2 + \left| [\delta y_\ell ]_{ \tau e} \right| ^2 } \le 109 . \end{aligned}$$
    (20)

  • \(\underline{\tau \rightarrow \mu \gamma {:}}\)

    $$\begin{aligned} \sqrt{ \left| [\delta y_\ell ]_{ \mu \tau } \right| ^2 + \left| [\delta y_\ell ]_{ \tau \mu } \right| ^2 } \le 8.7 . \end{aligned}$$
    (21)

Limits on the off-diagonal Yukawa couplings from their one-loop contribution to \(\ell _2 \rightarrow 3 \ell _1\) decays are weaker [29].

Finally, motivated by the constraints discussed above, we write the LFV Higgs branching fractions as

$$\begin{aligned} \mathrm{Br}(h \rightarrow \tau \mu )\approx & {} { | [\delta y_\ell ]_{\mu \tau } |^2 + | [\delta y_\ell ]_{\tau \mu } |^2 \over 2^2 } \times 1.5~\% , \nonumber \\ \mathrm{Br}(h \rightarrow \tau e)\approx & {} { | [\delta y_\ell ]_{e \tau } |^2 + | [\delta y_\ell ]_{\tau e} |^2 \over 100^2 } \times 18~\% , \\ \mathrm{Br}(h \rightarrow \mu e)\approx & {} { | [\delta y_\ell ]_{e \mu } |^2 + | [\delta y_\ell ]_{\mu e} |^2 \over 0.06^2 } \times 4 \times 10^{-9}. \nonumber \end{aligned}$$
(22)

We can immediately see that the indirect constraints allow for a sizable branching fraction of \(h \rightarrow \tau e\), and \(h \rightarrow \tau \mu \) decays. In particular, the percent-level branching fraction for \(h \rightarrow \tau \mu \), hinted at by the CMS excess, can be addressed in the EFT context without any tension with \(\tau \rightarrow \mu \gamma \) bounds. However, one should note that the \(\mu \rightarrow e \gamma \) constraint does not allow \(\mathrm{Br} (h \rightarrow \tau \mu )\) and \(\mathrm{Br} (h \rightarrow \tau e)\) to be simultaneously large. Observing both of these decays at the LHC would thus signify a breakdown of the EFT approach. On the other hand, \(\mathrm{Br}(h \rightarrow \mu e)\) is constrained to be small by the \(\mu \rightarrow e \gamma \) constraint, so as to be unobservable in practice.

There is also the question which explicit BSM models may generate the pattern of LFV Yukawa couplings required to produce \(\mathrm{Br} (h \rightarrow \tau \mu /e)\) at the level of a percent to per-mille. This turns out to be difficult in concrete models. Typically, satisfying all constraints is either completely impossible [30], or requires some fine-tuning and/or challenging model building [19, 31–42].

3.2 Flavor-changing top quark decays

Dimension-6 operators may also violate flavor in the quark sector. In the SM, quark flavor is not conserved due to off-diagonal CKM matrix elements, but flavor-changing neutral currents are forbidden at tree level. Therefore, the quark-flavor-violating processes involving the Higgs boson are suppressed by a loop factor, and in addition suppressed by the GIM mechanism. On the other hand, the couplings in Eq. (4) may lead to flavor-changing neutral currents at tree level.

From the experimental point of view, the most interesting of these processes are the ones involving the top quark. ATLAS and CMS have performed direct searches for Higgs-mediated flavor-changing neutral currents in top quark decays: \(t \rightarrow h q\), \(q = c, u\). Due to loop and GIM suppression, the branching fractions for these decays in the SM are prohibitively small. However, in models beyond the SM with new sources of flavor violation these decays are often enhanced to a level that may be observable at the LHC; see e.g. [43].

In the limit of massless charm or up quarks, the tree-level decay width is given by the formula:

$$\begin{aligned} \Gamma (t \rightarrow h q ) = {m_t^2 m_q \over 32 \pi v^2} \left( 1 - {m_h^2 \over m_t^2} \right) ^2 \left( | [\delta y_u]_{q t} |^2 + | [\delta y_u]_{t q} |^2 \right) . \end{aligned}$$
(23)

This translates to the branching fractions:

$$\begin{aligned} \mathrm{Br}(t \rightarrow h c)= & {} 1.1 \times 10^{-3} \left( | [\delta y_u]_{c t} |^2 + | [\delta y_u]_{t c} |^2 \right) , \nonumber \\ \mathrm{Br}(t \rightarrow h u)= & {} 1.9 \times 10^{-6} \left( | [\delta y_u]_{u t} |^2 + | [\delta y_u]_{t u} |^2 \right) , \end{aligned}$$
(24)

where we used \(\Gamma _t \approx 1.35\) GeV.

The current 95 % upper limits on the branching fractions for these decays are given in Table 2. Using these, we find the following constraints on the off-diagonal Higgs Yukawa couplings:

$$\begin{aligned} \sqrt{ | [\delta y_u]_{c t} |^2 + | [\delta y_u]_{t c} |^2}\le & {} 2.1, \nonumber \\ \sqrt{ | [\delta y_u]_{u t} |^2 + | [\delta y_u]_{t u} |^2}\le & {} 49. \end{aligned}$$
(25)
Table 2 List of experimental 95 % CL upper limits on the branching fraction \(\mathrm{Br}\) for Higgs-mediated quark-flavor-violating processes
Full size table

Much as for LFV Higgs decays to tau leptons, the current indirect constraints on \(\delta y_{qt}\) and \(\delta y_{tq}\) do not forbid the \(t \rightarrow h q\) branching fraction to be close to the current LHC limits. While the relative phase between \(\delta y_{qt}\) and \(\delta y_{tq}\) is severely constrained by neutron electric dipole moment searches [47], the absolute values (which enter into the \(t \rightarrow h q\) widths) are allowed to be large. One should also mention that D-meson oscillations place more severe constraints on the products \(\delta y_{ut} \delta y_{tc}\) and \(\delta y_{tu} \delta y_{ct}\); see [47]. Therefore, in the EFT context, it is impossible for both \(t \rightarrow h c\) and \(t \rightarrow h u\) branching fractions to be close to the current experimental limits.

4 Three-body decays

In the previous section we discussed two-body exotic decays induced by dimension-6 operators of the Yukawa type. We concluded that indirect constraints on the LFV and QFV Higgs Yukawa couplings to fermions are consistent with the branching fractions of \(h \rightarrow \tau \mu \) and \(h \rightarrow \tau e\) decays that are readily observable at the LHC. In fact, the best limits on the relevant couplings currently come from the LHC. This agrees with the conclusions from previous literature [14]. In this section we extend this discussion to three-body exotic Higgs decays and other operators appearing at the dimension-6 level in the EFT Lagrangian.

4.1 \(h \rightarrow W b q\)

We begin with the \(h \rightarrow t^* q \rightarrow W b q\) decays. These decays are mediated by the same Yukawa couplings as the ones leading to the \(t \rightarrow h c/u\) decays, and they are constrained by ATLAS and CMS searches as in Eq. (25):

$$\begin{aligned} \mathrm{Br} (h \rightarrow W b c)= & {} 1.3 \times 10^{-4} \left( | [\delta y_u]_{c t} |^2 + | [\delta y_u]_{t c} |^2 \right) , \nonumber \\ \mathrm{Br} (h \rightarrow W b u)= & {} 2.3 \times 10^{-7} \left( | [\delta y_u]_{u t} |^2 + | [\delta y_u]_{t u} |^2 \right) ,\nonumber \\ \end{aligned}$$
(26)

where we summed over the \(W^+\) and \(W^-\) modes. Note that Higgs decays with \(\mathcal{O}(10^{-4})\) branching fractions have already been seen in LHC Run-1 in the \(h \rightarrow ZZ \rightarrow 4 \ell \) channel. Thus, if \(t \rightarrow h q\) decays are observed at the LHC close to the current limit, it should be possible to also observe the \(h \rightarrow W b q\) decays in the future (although the \(t \bar{t}\) background will be a challenge in this case).

The same decay can also be mediated by dipole-type couplings in Eq. (6). Implementing the relevant vertices in FeynRules [48, 49] and calculating the decay width numerically in aMC@NLO [50] one finds

$$\begin{aligned} \begin{array}{ll} \mathrm{Br}(h \rightarrow W b c) &{}= 1.7 \times 10^{-4} \,| [d_{Wu}]_{c b} |^2 + 3.3 \times 10^{-7} \,| [d_{Wd}]_{s t} |^2 \, , \\ \mathrm{Br}(h \rightarrow W b u) &{}= 3.0 \times 10^{-7}\, | [d_{Wu}]_{u b} |^2 + 1.7 \times 10^{-8}\, | [d_{Wd}]_{d t} |^2, \end{array} \end{aligned}$$
(27)

where we sum over \(W^\pm \) decay modes. Barring fine-tuning, in the EFT approach the W-boson dipole couplings \(d_{Wq}\) are related to the analogous dipole couplings of the photon, \(d_{Wf} \sim s_\theta ^2 d_{Af}\). Given that, the parameters \(d_{Wd}\) entering the expressions above are subject to stringent constraints from B-physics, such as the measurement of the \(b \rightarrow s \gamma \) branching fraction, and thus their contribution to the Higgs decays is completely negligible. As for the \(d_{Wu}\) parameters, they are less severely constrained by the limits on \(\mathrm{Br}(t \rightarrow q \gamma )\) set by the CMS experiment [51], and we estimate \(| [d_{Wu}]_{c b} | \lesssim 0.06\), \(|[d_{Wu}]_{u b} | \lesssim 0.4\). This implies that the maximum branching fraction for dipole-mediated \(h \rightarrow W b q\) decays at most \(\sim 10^{-6}\), which is lower than the maximum Yukawa-mediated rate, and probably too low to be observable.

4.2 \(h \rightarrow \ell _1 \ell _2 \gamma \)

We move to the dipole-type operators in Eq. (6). In the lepton sector, these may lead to \(h \rightarrow \ell _1 \ell _2 \gamma \) decays, where the presence of a hard photon in the decay would allow experiments to distinguish it from \(h \rightarrow \ell \ell '\) mediated by Yukawa couplings.

As discussed before, the strength of Higgs dipole-type interactions is fixed by the strength of the corresponding dipole interaction between fermions and a gauge boson. Therefore the constraint on the Higgs coupling will come from dipole-mediated \(\ell _1 \rightarrow \ell _2 \gamma \) decays. In the limit where the leptons are massless, the width of the latter is given by

$$\begin{aligned} \Gamma (\ell _1 \rightarrow \ell _2 \gamma ) = {e^2 m_{\ell _1}^4 m_{\ell _2} \over 4 \pi v^4} \left( \left| [d_{Ae}]_{ \ell _1 \ell _2} \right| ^2 + \left| [d_{Ae}]_{ \ell _2 \ell _1} \right| ^2 \right) , \end{aligned}$$
(28)

where we summed over the \(\ell _1^+ \ell _2^-\) and \(\ell _1^- \ell _2^+\) decay modes. Using the experimental results from Table 1, we get the following constraints on the dipole couplings:

$$\begin{aligned} \sqrt{\left| [d_{Ae}]_{ e \mu } \right| ^2 + \left| [d_{Ae}]_{ \mu e} \right| ^2}\le & {} 1.2 \times 10^{-6}, \nonumber \\ \sqrt{\left| [d_{Ae}]_{ e \tau } \right| ^2 + \left| [d_{Ae}]_{ \tau e} \right| ^2}\le & {} 2.6 \times 10^{-3}, \nonumber \\ \sqrt{\left| [d_{Ae}]_{ \mu \tau } \right| ^2 + \left| [d_{Ae}]_{ \tau \mu } \right| ^2}\le & {} 2.1 \times 10^{-4}. \end{aligned}$$
(29)

The dipole-mediated \(h \rightarrow \ell _1 \ell _2 \gamma \) decay width is given by

$$\begin{aligned} \Gamma (h \rightarrow \ell _1 \ell _2 \gamma )= & {} {e^2 m_h^5 m_{\ell _1} m_{\ell _2} \over 384 \pi ^3 v^6}\nonumber \\&\times \left( \left| [d_{Ae}]_{ \ell _1 \ell _2} \right| ^2 + \left| [d_{Ae}]_{ \ell _2 \ell _1} \right| ^2 \right) , \end{aligned}$$
(30)

where we summed over the \(\ell _1^+ \ell _2^-\) and \(\ell _1^- \ell _2^+\) decay modes. Given Eq. (29), the branching fractions for dipole-mediated \(h \rightarrow \ell _1 \ell _2 \gamma \) decays are constrainedFootnote 3 as

$$\begin{aligned} \mathrm{Br}(h \rightarrow \mu e \gamma )\le & {} 1.9 \times 10^{-23}, \nonumber \\ \mathrm{Br}(h \rightarrow \tau e \gamma )\le & {} 1.7 \times 10^{-15}, \\ \mathrm{Br}(h \rightarrow \tau \mu \gamma )\le & {} 2.3 \times 10^{-15}.\nonumber \end{aligned}$$
(31)

Unlike for Yukawa-mediated two-body decays, this time the decays with \(\tau \) in the final states are constrained to be extremely rare. As long as the EFT framework is adequate for describing Higgs decays, there is no prospect of observing the dipole-mediated LFV decays at the LHC or the future 100 TeV collider [52].

4.3 \(h \rightarrow \ell _1 \ell _2 Z\)

Another process that can be generated by dipole-type interactions in Eq. (6) is \(h \rightarrow \ell _1 \ell _2 Z\). To calculate the width, we have implemented the \((1 + h/v) \bar{\ell }_1 \sigma ^{\mu \nu } \ell _2 V_{\mu \nu }\) vertex in FeynRules, and calculated the decay width numerically in aMC@NLO. We obtain the branching fractions:

$$\begin{aligned} \mathrm{Br}(h \rightarrow \mu e Z)= & {} 2.1 \times 10^{-12} \left( \left| [d_{Ze}]_{ e \mu } \right| ^2 + \left| [d_{Ze}]_{ \mu e} \right| ^2 \right) , \nonumber \\ \mathrm{Br}(h \rightarrow \tau e Z)= & {} 3.5 \times 10^{-11} \left( \left| [d_{Ze}]_{ e \tau } \right| ^2 + \left| [d_{Ze}]_{ \tau e} \right| ^2 \right) , \nonumber \\ \mathrm{Br}(h \rightarrow \tau \mu Z)= & {} 7.3 \times 10^{-9} \left( \left| [d_{Ze}]_{ \mu \tau } \right| ^2 + \left| [d_{Ze}]_{ \tau \mu } \right| ^2 \right) ,\nonumber \\ \end{aligned}$$
(32)

where we summed over the \(\ell _1^+ \ell _2^-\) and \(\ell _1^- \ell _2^+\) decay modes.

Table 3 Experimental \(95~\%\) CL upper limits on the branching fraction \(\mathrm{Br}\) for LFV Z boson decays
Full size table

Constraints on the parameters \(d_{Ze}\) come from experimental limits on LFV Z boson decays summarized in Table 3. The dipole mediated partial decay width is given by

$$\begin{aligned} \Gamma (Z \rightarrow \ell _1 \ell _2)= & {} {(g_L^2 + g_Y^2) m_Z^3 m_{\ell _1} m_{\ell _2} \over 6 \pi v^4} \nonumber \\&\times \left( \left| [d_{Ze}]_{ \ell _1 \ell _2} \right| ^2 + \left| [d_{Ze}]_{ \ell _2 \ell _1} \right| ^2 \right) , \end{aligned}$$
(33)

where we summed over the \(\ell _1^+ \ell _2^-\) and \(\ell _1^- \ell _2^+\) decay modes. This results in the following constraints on the dipole couplings:

$$\begin{aligned} \sqrt{\left| [d_{Ze}]_{ e \mu } \right| ^2 + \left| [d_{Ze}]_{ \mu e} \right| ^2}\le & {} 76 , \nonumber \\ \sqrt{\left| [d_{Ze}]_{ e \tau } \right| ^2 + \left| [d_{Ze}]_{ \tau e} \right| ^2}\le & {} 67, \\ \sqrt{\left| [d_{Ze}]_{ \mu \tau } \right| ^2 + \left| [d_{Ze}]_{ \tau \mu } \right| ^2}\le & {} 5.2.\nonumber \end{aligned}$$
(34)

Stronger constraints on these couplings are obtained through their loop contributions to radiative lepton decays [56, 57]. At one loop one finds

$$\begin{aligned} \Gamma (\ell _1 \rightarrow \ell _2 \gamma )= & {} {m_{\ell _1}^4 m_{\ell _2} e^2 m_Z^2(g_L^2 + g_Y^2) \over 1024 \pi ^5 v^6} \nonumber \\&\times \left( 3 - 6 c_\theta ^2 + 4 c_\theta ^2 \log c_\theta ^2 \right) ^2 \nonumber \\&\times \left( \left| [d_{Ze}]_{ \ell _1 \ell _2} \right| ^2 + \left| [d_{Ze}]_{ \ell _2 \ell _1} \right| ^2 \right) . \end{aligned}$$
(35)

Using the experimental results from Table 1, and assuming no cancellations between the tree-level \(d_{Ae}\) and the one-loop contribution from Z dipole, we get the following constraints on \(d_{Ze}\):

$$\begin{aligned} \sqrt{\left| [d_{Ze}]_{ e \mu } \right| ^2 + \left| [d_{Ze}]_{ \mu e} \right| ^2}\le & {} 2.7 \times 10^{-4}, \nonumber \\ \sqrt{\left| [d_{Ze}]_{ e \tau } \right| ^2 + \left| [d_{Ze}]_{ \tau e} \right| ^2}\le & {} 0.63, \\ \sqrt{\left| [d_{Ze}]_{ \mu \tau } \right| ^2 + \left| [d_{Ze}]_{ \tau \mu } \right| ^2}\le & {} 5.1 \times 10^{-2}.\nonumber \end{aligned}$$
(36)

This translates to the constraints on the branching fractions:

$$\begin{aligned} \mathrm{Br}(h \rightarrow \mu e Z)\le & {} 1.5 \times 10^{-19}, \nonumber \\ \mathrm{Br}(h \rightarrow \tau e Z)\le & {} 1.4 \times 10^{-11}, \\ \mathrm{Br}(h \rightarrow \tau \mu Z)\le & {} 1.9 \times 10^{-11}.\nonumber \end{aligned}$$
(37)

The suppression is slightly smaller than for the decays with a photon in the final state, however, observing decays with this low branching fraction is impossible at the LHC or at the 100 TeV collider.

The same process (though with a different helicity structure for the final-state fermions) can also be generated by vertex-type couplings in Eq. (5). Implementing the relevant vertices in FeynRules and calculating the decay width numerically in aMC@NLO one finds

$$\begin{aligned}&\mathrm{Br}(h \rightarrow \ell _1 \ell _2 Z)\nonumber \\&\quad \approx 7.0 \times 10^{-5}times \left( \left| [\delta g^{Ze}_L]_{ \ell _1 \ell _2} \right| ^2 + \left| [\delta g^{Ze}_R]_{\ell _1 \ell _2} \right| ^2 \right) .\nonumber \\ \end{aligned}$$
(38)

Again, the off-diagonal vertex corrections are constrained by LFV Z boson decays. The decay width is

$$\begin{aligned}&\Gamma (Z \rightarrow \ell _1 \ell _2 ) = {(g_L^2 + g_Y^2) m_Z \over 12 \pi }\nonumber \\&\quad \times \left( \left| [\delta g^{Ze}_L]_{ \ell _1 \ell _2} \right| ^2 + \left| [\delta g^{Ze}_R]_{\ell _1 \ell _2} \right| ^2 \right) . \end{aligned}$$
(39)

Then the experimental constraints in Table 3 imply:

$$\begin{aligned} \sqrt{ \left| [\delta g^{Ze}_L]_{e \mu } \right| ^2 + \left| [\delta g^{Ze}_R]_{e \mu } \right| ^2 }\le & {} 1.2 \times 10^{-3}, \nonumber \\ \sqrt{ \left| [\delta g^{Ze}_L]_{e \tau } \right| ^2 + \left| [\delta g^{Ze}_R]_{e \tau } \right| ^2 }\le & {} 4.3 \times 10^{-3}, \\ \sqrt{ \left| [\delta g^{Ze}_L]_{\mu \tau } \right| ^2 + \left| [\delta g^{Ze}_R]_{\mu \tau } \right| ^2 }\le & {} 4.8 \times 10^{-3}.\nonumber \end{aligned}$$
(40)
Table 4 Experimental \(90~\%\) CL upper limits on the branching fraction \(\mathrm{Br}\) for four-lepton flavor-violating processes
Full size table

Again stronger constraints arise through one-loop contributions to LFV lepton decays, for which the experimental limits are collected in Table 4. Assuming no cancellations with tree-level contributions of 4-fermion operators, one obtains the bounds [29]

$$\begin{aligned} \sqrt{ \left| [\delta g^{Ze}_L]_{e \mu } \right| ^2 + \left| [\delta g^{Ze}_R]_{e \mu } \right| ^2 }\le & {} 1.1 \times 10^{-6}, \nonumber \\ \sqrt{ \left| [\delta g^{Ze}_L]_{e \tau } \right| ^2 + \left| [\delta g^{Ze}_R]_{e \tau } \right| ^2 }\le & {} 5.9 \times 10^{-4}, \\ \sqrt{ \left| [\delta g^{Ze}_L]_{\mu \tau } \right| ^2 + \left| [\delta g^{Ze}_R]_{\mu \tau } \right| ^2 }\le & {} 4.1 \times 10^{-4}.\nonumber \end{aligned}$$
(41)

This translates to the following bounds on the branching fractions:

$$\begin{aligned} \mathrm{Br}(h \rightarrow \mu e Z)\le & {} 8.1 \times 10^{-17}, \nonumber \\ \mathrm{Br}(h \rightarrow \tau e Z)\le & {} 2.4 \times 10^{-11}, \\ \mathrm{Br}(h \rightarrow \tau \mu Z)\le & {} 1.2 \times 10^{-11}.\nonumber \end{aligned}$$
(42)

The bounds are somewhat weaker than for the dipole-mediated Higgs decays, however, the suppression is still too much for any realistic prospects of experimental detection.

4.4 \(t \rightarrow h q V\)

Finally, we consider flavor-violating three-body decays of the top quark mediated by dipole-type operators: \(t \rightarrow h q V\), where V is a photon or a gluon. We have implemented the \(\bar{t} \sigma ^{\mu \nu } c V_{\mu \nu }\) and \(h \bar{t} \sigma ^{\mu \nu } c V_{\mu \nu }\) vertices in FeynRules, and calculated the decay width numerically in aMC@NLO. We find

$$\begin{aligned} { \mathrm{Br}(t \rightarrow q V h) \over \mathrm{Br}(t \rightarrow q V) } \approx 4.4 \times 10^{-8}. \end{aligned}$$
(43)

The current best constraints on \(\mathrm{Br}(t \rightarrow q \gamma )\) come from searches for anomalous top production at the LHC. For the dipole couplings to photons the strongest limits come from the CMS experiment [51]. They translate to the following limits on the branching fractions:

$$\begin{aligned} \begin{array}{lll} \displaystyle \mathrm{Br}(t \rightarrow u \gamma ) \le 1.3 \times 10^{-4} , \\ \displaystyle \mathrm{Br}(t \rightarrow c \gamma ) \le 1.7 \times 10^{-3} . \end{array} \end{aligned}$$
(44)

For \(t \rightarrow u \gamma \), even stronger limits can be placed due to the dipole contributions to the neutron electric dipole moment [60], though these constraints do no apply when the dipole couplings are parity conserving. For the dipole couplings to the gluon, the strongest limits come from the ATLAS experiment [61]:

$$\begin{aligned} \begin{array}{lll} \displaystyle \mathrm{Br}(t \rightarrow u g) &{}\le &{} 4.0 \times 10^{-5} , \\ \displaystyle \mathrm{Br}(t \rightarrow c g) &{}\le &{} 1.7 \times 10^{-4} . \end{array} \end{aligned}$$
(45)

Limits on the flavor-violating dipole top couplings from \(t \gamma \) production at the LHC [62] are currently weaker. The experimental bounds in Eqs. (44) and (45) translate to the following constraints on dipole-mediated top decays with the Higgs:

$$\begin{aligned} \begin{array}{lll} \displaystyle \mathrm{Br}(t \rightarrow u \gamma h) &{} \le &{} 5.7 \times 10^{-12}, \\ {} \displaystyle \mathrm{Br}(t \rightarrow c \gamma h) &{} \le &{} 7.5 \times 10^{-11}, \\ {} \displaystyle \mathrm{Br}(t \rightarrow u g h) &{} \le &{} 1.8 \times 10^{-12}, \\ {} \displaystyle \mathrm{Br}(t \rightarrow c g h) &{} \le &{} 7.5 \times 10^{-12} . \end{array} \end{aligned}$$
(46)

As in the case of \(h \rightarrow \ell ^+_1 \ell ^-_2 \gamma \) decays, the branching fraction for \(t \rightarrow q \gamma /g h\) may be larger than the limits in Eq. (46) if the process is mediated by off-diagonal Yukawa couplings and the photon or gluon is emitted from the final state quark. Although the top production cross section is larger than that of the Higgs boson (\(\mathcal{O}(1)\) nb at the 14 TeV LHC, and a factor of 30 larger at 100 TeV, which corresponds to \(\mathcal{O}(10^{11})\) \(t\bar{t}\) pairs in the future 100 TeV collider [52]), the limits in Eq. (46) leave little room for observing the dipole-mediated top quark decays. A more realistic probe of flavor-violating dipole-type Higgs interactions may be offered by the production processes \(p p \rightarrow h t j\) and \(p p \rightarrow h t \gamma \).Footnote 4

5 Conclusions

In this paper we discussed the possibility of observing at the LHC exotic Higgs decays that violate lepton or quark flavor. Our study was done in the context of an EFT which describes the effective interactions of the Higgs boson with other SM particles after heavy particles from beyond the SM have been integrated out. In this context, the possibility of a significant rate of two-body decays such as \(h \rightarrow \mu \tau \), \(h \rightarrow e \tau \), and \(t \rightarrow c h\) was pointed out in the previous literature. Our analysis confirms and updates these conclusions.

We also studied the possibility of exotic three-body decays involving the Higgs boson. Here, our conclusions are largely negative. The existing precision constraints imply that the rate of such three-body processes must be prohibitively small and cannot be observed in colliders in the foreseeable future. This is an important and robust conclusion that can derived in the EFT framework. Conversely, if such three-body processes are observed, this would signal a breakdown of the EFT description we used regarding Higgs decay processes. Such a breakdown would be a harbinger of new light degrees of freedom, or a non-linear realization of electroweak symmetry.