Explaining Atomki anomaly and muon $g-2$ in $U(1)_X$ extended flavour violating two Higgs doublet model

We investigate a two Higgs doublet model with extra flavour depending $U(1)_X$ gauge symmetry where $Z'$ boson interactions can explain the Atomki anomaly by choosing appropriate charge assignment for the SM fermions. For parameter region explaining the Atomki anomaly we obtain light scalar boson with $\mathcal{O}(10)$ GeV mass, and we explore scalar sector to search for allowed parameter space. We then discuss anomalous magnetic dipole moment of muon and lepton flavour violating processes induced by Yukawa couplings of our model.


I. INTRODUCTION
The standard model (SM) of particle physics has been very successful in accommodating experimental data. On the other hand the SM is not believed to be complete theory and existence of physics beyond the SM has been discussed. One of the minimal extensions of the SM is introduction of new gauge symmetry such as extra U (1). Another possibility is the extension of Higgs sector introducing second Higgs doublet field so called two Higgs doublet model (THDM). Such extensions give possibilities to explain some observed anomalies which would not be explained within the SM.
Muon anomalous magnetic dipole moment (muon g − 2) is known as a long-standing anomaly which is discrepancy between the standard model (SM) prediction and observation [31]; ∆a µ = a exp µ − a SM µ = (26.8 ± 6.3 ± 4.3) × 10 −10 . (I.1) where the deviation from the SM prediction is ∼ 3σ level with a positive value; recent theoretical analysis further indicates 3.7σ deviation [32] and other analysis is also found in refs. [33][34][35][36]. Moreover, several upcoming experiments such as Fermilab E989 [37] and J-PARC E34 [38] will give the results with more precision. Although the recent result on the hadron vacuum polarization (HVP), calculated by Budapest-Marseille-Wuppertal (BMW) collaboration [40], weakens the necessity of a new physics effect, it is shown in refs. [41,42] 1 that the BMW result indicates new tensions with the HVP extracted from e + e − data and the 1 The effect in modifying HVP for muon g − 2 and electroweak precision test is previously discussed in ref. [43]. global fits to the electroweak precision observables. If the anomaly is confirmed, the muon g −2 is a clear signal of a new physics effects; various solutions have been proposed to explain the anomaly such as scenarios shown in refs. . One of the attractive solution is given in general THDM where lepton flavour violating interactions provide sizable contribution.
In this paper we consider a model with extra U (1) X gauge symmetry and two Higgs doublets where the Atomki anomaly and muon g −2 can be explained. The Atomki anomaly is explained by Z boson from flavour dependent U (1) X gauge symmetry introduced in ref. [14] which also induce relatively light scalar boson with mass of O(10) GeV or less from a SM singlet scalar breaking U (1) X symmetry spontaneously. We then discuss muon g − 2 and lepton flavour violation(LFV) processes originated from Yukawa interactions in THDM sector. In addition neutrino mass can be generated by introducing right-handed neutrino without U (1) X charge keeping anomaly cancellation condition; neutrino physics is often discussed in extra U (1) models [56,58,[70][71][72][73][74][75][76][77][78]. Taking into account light scalar boson associated with U (1) X breaking, we numerically investigate scalar potential as well as muon g − 2 and LFV constraints searching for allowed parameter region.
This paper is organized as follows. In sec. II, we show our model and mass spectrum of new particles. In sec. III, we discuss Z interactions explaining the Atomki anomaly. In sec. IV, we discuss scalar potential in the model. In sec. V, we discuss muon g − 2 and LFV constraints. In sec. VI, we briefly discuss dark sector for realizing quark mixing under flavour dependent U (1) X . Our summary is given in Sec. VII.

II. MODEL
In this section we review our model. We consider a family non-universal U (1) X charge assignments for fermions in general two Higgs doublet model as shown in Table I. If no discrete Z 2 symmetry is imposed, both the Higgs doublets can couple to all the fermions depending on U (1) X charge assignment. In the flavour eigenstates of the fermions, the Yukawa Lagrangian is written by whereH 1,2 ≡ iσ 2 H * 1,2 with σ 2 being the second Pauli matrix. The structure of Yukawa matrices are fixed by assignment of U (1) X charge to fermions and Higgs fields. In the scalar sector we introduce the SM singlet φ which develops vacuum expectation value(VEV), v φ , to break U (1) X gauge symmetry. The Higgs doublets and singlet scalar are represented as where v i and v φ are the vacuum expectation values(VEVs) of the Higgs doublet H i and φ We analyze scalar potential after discussing U (1) X charge assignment below.
A. Constraints for U (1) X charge assignment Here we consider constraints for U (1) X charge assignment such as conditions to obtain the SM fermion masses and anomaly cancelation conditions associated with U (1) X and the SM gauge symmetries as discussed in ref. [14]. To obtain fermion masses, we first impose the constraints on the charges to realize the diagonal Yukawa coupling such that By these conditions, we can obtain diagonal elements of Yukawa matrices and masses of the SM fermions can be realized. We then further discuss necessary constraints for the charge assignment.
The conditions for canceling gauge anomalies are given by Combining the anomaly cancellation conditions and the constraints in Eq. (II.3) we obtain Note that charges of two Higgs doublets are required to be identical when we require both Higgs doublets to couple with fermions satisfying anomaly cancellation conditions. We then write Q H ≡ Q H 1 = Q H 2 hereafter. Now U (1) X charges can be written in terms of Q H , Q Q 2 , In our scenario we require universal charge assignment in lepton sector, which is preferred to suppress interactions between neutrinos and Z boson. Then we impose and this implies In the universal lepton charge assignment both H 1 and H 2 can have Yukawa interaction inducing flavour changing neutral current (FCNC) interactions. We also require the relation (II.14) The relation makes Z coupling to d-and s-quark to be the same realizing minimal flavour violation as we obtain In this case we get the other charges in terms of two independent charges Q H and Q Q 12 , given below.
Notice that in Eq. (II.16) we can assign universal charges to the quark sector by consid- . Actually in this simple case, the charges are proportional to U (1) Y charge assignment such that where it is the same as hypercharge when we choose Q H = 1/2 with a different gauge coupling. In fact, ratio of charges are relevant and the common factor can be absorbed in normalizing g .

B. Gauge Sector
Here we discuss gauge sector of the model. The most general Lagrangian for U (1) gauge sector including the kinetic mixing term is written by where B µν and B µν are the field strength tensors of U (1) Y and U (1) X gauge symmetries. Here x(<< 1) is a dimensionless gauge kinetic mixing parameter. We can diagonalize Eq. (II.18) by the following transformation Under the transformation Eq. (II. 19), the gauge Lagrangian can be written as The kinetic terms of the scalar fields are where, After scalar fields developing their VEVs we obtain mass terms for gauge fields, and those of neutral gauge fields (W 3 µ ,B µ ,B µ ) are (II.23) The mass matrix M 2 gauge is written by where we parameterize Here we rotate (W 3 ,B) by Weinberg angle such that Then we obtain massless photon field as in the SM and the mass matrix for massive components is The physical masses of the neutral gauge bosons are The mass matrix in Eq. (II.27) can be diagonalized by rotation matrix where Z µ and Z µ are mass eigenstates for the SM Z and extra Z bosons. In summary, the original gauge fields are transformed into mass eigenstates by where a(b) = 1, 2, 3.

C. Scalar sector
Here we discuss scalar sector under our U (1) X charge assignment formulating mass spectrum and corresponding mass eigenstate. The scalar potential in the model is written as where we choose all the couplings to be real for simplicity. The VEVs can be obtained by solving the conditions ∂V /∂v 1 = ∂V /∂v 2 = ∂V /∂v φ = 0 which require the parameters to whereλ = λ 3 + λ 4 + λ 5 and for simplicity we consider λ 6 , λ 7 and λ 10 to be zero.
The mass eigenstates for charged components are obtained as physical charged Higgs boson. The mass of charged Higgs boson is given by The mass eigenstates for CP-odd neutral scalar bosons can be given by where G 0 1 and G 0 2 are two goldstone modes absorbed by Z and Z bosons, and A 0 is massive CP-odd scalar. Then mass of A 0 is given by The CP-even scalar sector has three physical degrees of freedom {h 1 , h 2 , φ R } and the mass matrix is given by The mass matrix can be diagonalized by an orthogonal matrix R with three Euler parameters and mass eigenstates are obtained such that (II.41)

D. Quark masses and Yukawa interactions
Here we consider Yukawa couplings for quarks Under our U (1) X charge assignment structure of Yukawa matrices are where "×" indicates non-zero component. In this case we cannot obtain realistic quark mixing at renormalizable level. We thus assume contributions from higher order operators where a = 1, 2 and we assume U (1) X charge of φ to be Q φ = 3Q 12 − Q H to make these terms gauge invariant. We will discuss possible hidden sector realizing such non-renormalizable term in sec. VI.
After electroweak symmetry breaking, the mass terms of the quark sector are Then quark mass matrices are given by The mass matrices can be diagonalized by unitary matrices connecting flavour and mass eigenstates as u L,R = V u L,R u L,R and d L,R = V d L,R d L,R where we write primed field as mass eigenstates.
Here we write Yukawa interactions in terms of mass eigenstates and the quark Yukawa for up-type quarks and for down-type quarks. Here the interaction matrix X u,d are defined by

E. Lepton masses and Yukawa interactions
Here we consider Yukawa couplings for leptons Under our U (1) X charge assignment leptons have flavour-universal charges and all the elements in Y l 1,2 can be non-zero at renormalizable level. After electroweak symmetry breaking we obtain charged lepton mass term such as The mass matrix can be diagonalized by biunitary transformation by defining the physical fields (mass eigenstates) of leptons as l L,R = V l L,R l L,R , and diagonal mass matrix is given by The Yukawa interactions among neutral scalar and charged leptons can be written by where coupling matrix X l is defined as Note that X l is not diagonal in general which induce flavour violating processes which will be discussed below.
Finally, we consider the leptonic interaction with charged Higges, where mass eigenstate of the charged Higgs is given by Eq. (II.35). In this paper, we consider neutrino mass is generated introducing right-handed neutrinos ν R which are gauge singlet.
Then we can apply type-I seesaw mechanism where detailed analysis is omitted. The physical ν L is related to the unphysical ν L by The leptonic interaction with charged Higgs takes the form where, Writing the Yukawa interaction terms of the leptons, to the scalar, we get where φ = h 0 , H 0 , ξ 0 , A 0 and h 0 ,H 0 ,ξ 0 = 1, A 0 = i. Also Y l φ can be obtained from Eq. II.52.

III. Z INTERACTION WITH SM FERMIONS AND ATOMKI ANOMALY
In this section, we discuss Z interactions associated with the SM fermions and possibility to explain Atomki anomaly.
Quark Sector: Kinetic terms for quarks are where the superscrript "0" indicates the fermions in flavour eigenstate. The covariant derivatives are given by Lepton Sector: Kinetic terms for leptons are For discussing explanation of Atomki anomaly we focus on Z interactions associated with first generation of fermions. The Z µ coupling to first generation of quarks are given by Applying Eq. (II.32) and (II.31), the coupling coefficients are given by Similarly, the Z µ coupling to first generation of leptons are given by where the relevant coefficients are The interaction between SM fermions and Z µ can be written as where f = {u, d, e, ν e } distinguishes type of fermion. Thus we have the relations A g sin θ w (III.11) To explain Atomki anomaly we require the constraints [8,9]; where the nucleon coupling to The Z µ coupling needs to be protophobic and satisfies the relation Here we assume the condition is at least satisfied approximately by tuning our free parameters. Furthermore, the charge assignment conditions in Eq. (II.16) and Eq. (III.14) makes all the axial couplings of Z µ to be zero. Using the charges in Eq. (II.16), we get We define the parameters, C p V = δ, C n V = and C e V = κ. Then κ = sin θ cos θ w sin θ w δ = − 1 2 sin θ sin θ w cos θ w + sin θ w cos θ w sin θ + 1 2 ρ cos θ cos θ w − 3g cos θ ρ x Q Q 12 = 1 2 sin θ sin θ w cos θ w + 1 2 ρ cos θ cos θ w − 3g cos θ ρ x Q Q 12 (III.16) whereg = g /e = g /g sin θ w . In terms ofg , Eq. (III.14) can be written as 1 2g The parameters κ, δ and are related as Here we comment on the case of flavour independent charge assignment Q Q 12 = Q H /3 as given in Eq. (II.17). In this case, the vector and axial couplings in Eq. (III.6) become Then it is not possible to satisfy Eq. (III.12).  [84] . We also show, in Fig. 2, resultant |κ|, |δ| and | | for parameters satisfying the Atomki constraints and NA64 limit for g . We find that sin θ < 10 −3 and it can be allowed by electroweak precision measurement [85]. Since typical value of g is O(10 −3 ) the typical VEV of φ is estimated as m Z ( 17MeV)/g ∼ O(10) GeV. In the following scalar potential analysis we apply v φ = 10 GeV as a reference value.
Note that we have quark flavour changing interactions associated with Z since U (1) X charge assignment for quarks is flavour dependent where the interactions are suppressed by CKM elements [14]. In addition such interactions are also induced by Yukawa interactions

IV. ANALYSIS OF SCALAR POTENTIAL
In this section, we analyze scalar potential of the model which includes two Higgs doublet and new scalar to break U (1) X gauge symmetry. We write parameters in scalar potential {λ 1 , λ 2 , λ 3 , λ 4 , λ 11 , λ 8 , λ 9 } by physical masses and VEVs such that The constraints from unitarity and perturbativity are given by [82,83] |λ 1,2,3,11 | ≤ 4π, |λ 8,9 | ≤ 8π, |λ 3 ± λ 4 | ≤ 8π, |λ 3 + 2λ 4 + 3λ 5 | ≤ 8π, where a 1,2,3 are the solution of the following equation We adopt the conditions for vacuum stability [82]: where we assume λ 4 ≤ λ 5 by choosing m A 0 < m H ± . In addition, we consider constraints from h 0 → Z Z decay where the decay width of the process is approximately given by Since Z is very light as m Z ∼ 17 MeV it is difficult to see decay products from the decay chain, and we consider the mode contributes to Higgs invisible decay mode. In addition, we consider h 0 → ξ 0 ξ 0 decay which also contribute to Higgs invisible decay mode since ξ 0 mainly decays into Z Z pair. The decay width is given by Then we apply constraint from invisible decay of the SM Higgs for the branching ratio of the process [100] In addition to the invisible decay BR, the other SM Higgs signal measurements also constrain the mixings among scalar bosons since it induces deviation of the SM Higgs couplings [99,100]. In our analysis we apply HiggsSignals-2.2.3beta [98] code to impose the constraints from the Higgs signal strength data and to obtain the 2σ allowed parameter space.
Note also that we can consider flavour constraints such as b → sγ induced from quark Yukawa couplings. In this paper, we do not discuss such constraints and we assume quark Yukawa couplings are tuned to satisfy all the quark flavour constraints. For lepton Yukawa couplings, we discuss constrains from flavour violation along with muon g − 2 in Sec. V.
Here we scan out parameters to search for allowed parameter region, such that v φ = 10 GeV is applied as indicated by solving Atomki anomalies, and we assume m A 0 < m H 0 for simplicity. Further, m H 0 = m H ± is assumed to avoid constraints from the T -parameter.
We show the allowed parameter regions in Fig. 3 for mixing angles, tan β and scalar masses.
It is then found that α 1 > 0 is preferred and there is correlation between tan β and α 1 .
Furthermore α 2 and α 3 should be small to suppress BR of h 0 → Z Z and h 0 → ξ 0 ξ 0 processes. The mass of ξ 0 is found to be less than ∼ 35 GeV due to small VEV of φ and perturbativity constraints for couplings in scalar potential. We also show correlations among scalar masses for allowed parameter region in Fig. 4. For m A 0 < m h 0 /2, the parameter space gets strongly constrainted as the decay width of h 0 → A 0 A 0 becomes significantly high. Detailed discussions regarding extra Higgs boson signals can be referred to e.g. refs. [86][87][88][89][90][91][92][93][94][95][96][97]. Note also that H 0 and A 0 can decay into ξ 0 ξ 0 mode since it tends to be light. As ξ 0 dominantly decays into Z Z we would have decay chain of H 0 (A 0 ) → ξ 0 ξ 0 → 4Z where Z decays into the SM fermions. Since Z is so light as ∼ 17 MeV to explain Atomki anomaly it is very challenging to identify such Z at detectors of the LHC. We thus do not discuss such signals although it will be the unique signatures of our scenario.

V. MUON g − 2 AND LEPTON FLAVOUR VIOLATION
In this section, we discuss muon g − 2 and constraints from LFV processes induced by Yukawa interactions in Eq. (II.52).
In our model we obtain one loop diagrams contributing to muon g − 2 in which neutral scalar boson and charged leptons propagating. Among them diagrams including τ inside loop would be dominant since they have enhanced by m τ via chiral flip, and they are proportional to X 23 X 32 factor. Then the muon g − 2 contribution from such diagrams is where explicit forms of Z φ are given by Here we neglect all the terms quadratic in muon mass as they are subdominant, hence the contribution from charged Higgs interactions is neglected. In addition, we assume scalar boson masses are sufficiently heavier than charged lepton masses.
Since muon g − 2 is explained by flavour violating coupling X 23,32 we need to take into account constraints from LFV processes. Firstly we consider h 0 → µτ decay, and the branching ratio is given by where m h 0 = 125 GeV and we assume Γ h 0 = 4.21 MeV.
Secondly we consider τ → 3µ process. The branching ratio of the process is given by where τ τ is the lifetime of τ .
Thirdly we consider µ → eγ process where the effective Lagrangian can be written as The ratio of the muon branching ratios given by where G F is the Fermi constant. The coefficients are sum of contributions from scalar boson loop diagrams where the explicit forms are given by and Here we also show approximated forms assuming scalar bosons are sufficiently heavier than charged lepton masses.
Finally we consider τ → µγ process. The relevant effective Lagrangian is where the Wilson coefficients are sum of contributions from the one-loop neutral and charged Higgs bosons The explicit forms of the coefficients are given by and where we assume scalar boson masses are sufficiently heavier than charged lepton masses.
In addition we also include the two-loop contributions given by Then the branching ratio for τ → µγ is expressed as In our analysis we consider (1) symmetric case X l 23 = X l 32 and (2) anti-symmetric case X l 23 = −X l 32 in estimating muon g − 2 and LFV processes. We then apply parameter range the same as shown in Eq. (IV.18) for scalar masses (except for lower limit of ξ 0 mass) and mixings, and X ij ∈ [10 −5 , 1] for Yukawa couplings. The lower limit of m ξ 0 is taken to be 10 GeV so that we can apply approximated form of LFV formulae 3 .
(1) Symmetric case X l 23 = X l 32 : The left panel of Fig. 5 shows the dependence of X l 23 with ∆a µ . We can get sizable contribution to muon g − 2 for X l 23 10 −4 . On the right panel of the figure, the allowed region of X l 23 -tan β parameter space is shown with the value of parameter α 2 indicated by color gradient. We can clearly see the correlation of X l 23 with tan β and α 2 . For X l 23 10 −2 , sizable muon g − 2 can be obtained only for large tan β and large α 2 . This behavior can be understood by the fact that the dominant contribution to Z φ in Eq. (V.2) comes from the ξ 0 term and small X l 23 requires large tan β and large α 2 since R 13 = sin α 2 . The contribution to ∆a µ from the pseudoscalar A 0 is negative in this case. Thus it is preferred that A 0 is heavier than the other neutral scalar bosons so that negative contribution is relatively small 4 . µτ ) < 0.25% [101] puts a very strong constraint on the allowed range of X l 23 . In fact, the constraint of BR(h 0 → µτ ) restricts X l 23 0.003 for tan β < 1 and the upper limit of X l 23 is ∼ 1.5 × 10 −3 for tan β > 1 . Apart from that, there is a correlation between tan β and α 1 which should be consistent with Fig. 3. The same allowed region of X l 23 − tan β can be obtained for X l 22 ranging from (10 −4 − 10 −1 ). The allowed region satisfying BR(µ → eγ) constraints for X l 13 in the range (10 −8 − 10 −2 ) as shown by the colorbar. For X l 33 in the range (10 −3 − 10 −1 ), as shown in the colorbar, we can satisfy the BR(µ → eγ) constraint.
Comparing Fig. 5 and Fig. 6, there is a narrow strip of parameter space satisfying the LFV constraints and the muon g − 2 constraint around X l 23 ∼ 10 −4 − 10 −3 . Fig. 7 shows the allowed region of parameter space consistent with the LFV constraints and muon g − 2 constraint. For the allowed parameter space shown in the left panel, the X l 23 − tan β space is allowed for X l 23 in the range (10 −4 − 1.5 × 10 −3 ) and tan β 3. In the right pannel the allowed region in α 2 − α 3 space is shown where | α 2 | /π 0.02 is more dominant and is consistent with the right panel of Fig. 5.
(2) Anti-symmetric case X l 32 = −X l 23 : Here we estimate muon g − 2 and LFV constraints for anti-symmetric Yukawa coupling case of X l 32 = −X l 23 as in the symmetric case. We find that the parameter space can be relaxed in this case since the muon g − 2 gets positive contribution from A 0 loop for the entire parameter space. In this antisymmetric situation the dominant contribution to ∆a µ comes from A 0 , and hence α 2 can be sufficiently low for X l 23 10 −4 as shown in the right panel of Fig. 8. The left panel of Fig. 8 shows a clear improvement in the range of X l 23 allowed by the muon g − 2 constraint. We get sizable contribution for X l 23 above 10 −4 . We also find that the muon g − 2 condition requires relatively large value for tan β for small X l 23 to enhance A 0 contribution. For LFV constraints we obtain the same behavior as the symmetric case shown in Fig. 6 and parameter space explaining muon g − 2 still can satisfiy the LFV constraints by proper choice of X l 22 , X l 13 and X l 33 . In Fig. 9, we show parameter space explaining muon g − 2 and satisfying LFV constraints at the same time. Comparing the left panels of Fig. 7 and Fig. 9, we see that a larger region of X l 23 − tan β space is allowed for the antisymmetric case. Finally, as mentioned above, α 2  can be sufficiently low since A 0 contribution is not suppressed by small α 2 . Thus parameter space explaining muon g −2 is qualitatively different in symmetric and anti-symmetric cases.

VI. DISCUSSION: DARK SECTOR FOR REALISTIC QUARK YUKAWA
In this section, we briefly discuss possible dark sector to induce Yukawa terms in Eq. (II.44) which are absent at tree level due to our U (1) X charge assignment. As a dark sector, for example, we introduce Z 2 odd fields as follows: vector like up-type quark U which is SU (2) singlet and U (1) X charge is 2Q H − 2Q 12 , vector like down-type quark D which is SU (2) singlet and U (1) X charge is −Q H + Q 12 , inert Higgs doublet η which has U (1) X is 2Q H − 3Q 12 and inert singlet scalar χ without U (1) X charge. Then we can write additional Lagrangian terms for quark Yukawa generation such that where a = 1, 2 and we omit terms irrelevant to induce quark Yukawa. From these interactions, we obtain the first two terms and the last two terms in Eq. (II.44) at one-loop level as shown in Fig. 10. We can also extend dark sector and provide other terms of Eq. (II.44).
In principle, we can tune free parameters to satisfy quark mixing while avoiding flavour constraints. Interestingly dark sector includes dark matter candidates which are neutral component of η and χ. In this paper we omit detailed analysis including dark matter physics and flavour constraints for dark sector since they are beyond our scope. Then we have investigated scalar potential which contain two Higgs doublet and one singlet. Taking into account small VEV of the singlet, we have searched for allowed parameters such as scalar boson masses and mixings. For scalar mixing, we have found stringent constraints come from the SM Higgs decay into Z Z and light scalar pair ξ 0 ξ 0 .
We also discussed muon g − 2 and LFV constrains in our model. The observed muon g − 2 can be explained by flavour violating Yukawa couplings in general two Higgs doublet model due to m τ enhancement from chiral flip inside an one loop diagram. We investigate constraints from LFV process associated with the flavour violating couplings to explain muon g − 2. In addition, effects of light scalar has been considered which provide changes from pure two Higgs doublet results.
Finally, we have discussed possible dark sector which can realize realistic quark mixing in our model where mixings associated with third generation quark is absent at renormalizable Lagrangian level due to our charge assignment for explaining the Atomki anomaly. By introducing dark sector, we can generate such mixings at one loop level and realize observed CKM matrix in principle. We have not discuss effect of dark sector in detail and it will be given in future works.