A singlet doublet dark matter model with radiative neutrino masses

We present a detailed study of a combined singlet-doublet scalar and singlet-doublet fermion model for dark matter. These models have only been studied separately in the past. We show that their combination allows for the radiative generation of neutrino masses, but that it also implies the existence of lepton-flavour violating (LFV) processes. We first analyse the dark matter, neutrino mass and LFV aspects separately. We then perform two random scans for scalar dark matter imposing Higgs mass, relic density and neutrino mass constraints, one over the full parameter space, the other over regions where scalar-fermion coannihilations become important. In the first case, a large part of the new parameter space is excluded by LFV, and the remaining models will be probed by XENONnT. In the second case, direct detection cross sections are generally too small, but a substantial part of the viable models will be tested by future LFV experiments. Possible constraints from the LHC are also discussed.


Introduction
While the Standard Model (SM) is certainly a highly successful theory of particle physics and, with the discovery of a Higgs-like boson by the ATLAS [1] and CMS [2] experiments at the CERN Large Hadron Collider (LHC) in 2012, is sometimes presented as complete, it is generally believed to suffer from conceptional and phenomenological deficiencies. Important examples are the unexplained large hierarchy of interactions [3][4][5][6], the surprisingly small, but non-zero masses of neutrinos [7] established first in 1998 by the oscillations of atmospheric [8], then of solar [9,10] and reactor neutrinos [11], and the observational evidence from many different scales [12] for a sizeable dark matter density in the Universe, quantified by WMAP [13] and then, more precisely, by the Planck satellite [14] to be Ω obs h 2 = 0.1186± 0.0031. Here, h denotes the present Hubble expansion rate in units of 100 km s −1 Mpc −1 .
An intriguing remedy for some of these problems is to extend the SM minimally by a small number (≤ 4) of new scalar and fermion fields, which allow to generate small neutrino masses radiatively and which have one (or more) neutral components that could represent dark matter. One of the most popular so-called radiative seesaw models is the scotogenic model with only one additional scalar (inert Higgs) SU(2) L doublet and a (righthanded neutrino) fermion singlet [15], for which we recently demonstrated the importance of coannihilations between the scalar dark matter and the right-handed neutrinos [16]. Many variants with general N -tuplets containing in general two scalars and one fermion have subsequently been proposed [17][18][19][20]. The new fields are usually assumed to be odd under a Z 2 symmetry to stabilize the dark matter and (in some cases) to prevent tree-level contributions to neutrino masses. All one-loop models connecting neutrino masses to dark JHEP10(2018)055 matter with at most four additional fields have been classified [21] following the notation of a systematic study of the d = 5 Weinberg operator at one-loop order [22]. The classification has recently been extended to two loops [23] following ref. [24] (cf. also section 4.7 in [25]).
Apart from the models mentioned above, which all belong to the one-loop topology T3 with one quartic scalar vertex as defined in refs. [21,22] and as shown in figure 1 (lower right), several models with other topologies have also been studied in the past. In addition to the topology label X in TX-Y, they have been labeled -when applicable -according to the number of internal fermion lines Y. They include models T1 with box topology such as a model of topology T1-1 with three scalars, two of which are equivalent, and one fermion (figure 1, upper left) [26], and a model of topology T1-3 with one scalar and three fermions (figure 1, lower left) [27][28][29]. In both cases, the lightest scalar was assumed to represent the dark matter, constraints from the Higgs and neutrino sectors and the relic density were imposed, and their phenomenology, in particular of lepton-flavour violating processes, has been studied. In this paper, we study a model of topology T1-2 with two scalars and two fermions (figure 1, upper right). A general discussion of this topology, without establishing the particle content of specific models, has been presented in ref. [30]. While ref. [22] has identified additional topologies denoted T2, T4, T5 and T6, we do not concentrate on them here. Notably, T2 was already immediately dismissed there because it involves mass-suppressed quartic vertices of two fermions and two scalars, while models of T4, T5 and T6 always have tree-level contributions to neutrino masses and thus no obvious connection to dark matter. In addition, three out of six models of type T4 and all models of type T5 and T6 are divergent at one loop and would require ultraviolet completion.

JHEP10(2018)055
As mentioned above, we study in this paper a model of topology T1-2 with two scalars and two fermions. We focus on the first of these models, T1-2-A in the classification of ref. [21], with a singlet Majorana fermion ψ S of hypercharge parameter α = 0, that determines also the hypercharges of the vector-like fermion doublet ψ D , the singlet scalar φ S , and the complex doublet scalar φ D . The alternative case α = −2, called the Inert Zee Model, has previously been analysed and found to be consistent with constraints from neutrino oscillation, dark matter and lepton-flavour violation data [31]. A doublet-triplet model, T1-2-F with α = −1, has also been analysed in a similar way [32]. Since our model T1-2-A with α = 0 is equivalent to T1-2-C with α = −1, our results equally apply to this model. It is furthermore one of the models, together with the model T1-3-A and α = 0 [27][28][29], that allows for gauge coupling unification at a scale Λ = O(10 13 ) GeV, at variance with most of the other models mentioned above including the original scotogenic model [33].
The paper is organised as follows: in section 2 we define our model by describing its particle content, mass and interaction Lagrangians, free parameters and particle mixings. In section 3 we review the implications for the dark matter relic density first of the scalar and fermion sectors separately, before we study the new regions of parameter space that open up when they couple to each other as required for neutrino mass generation. The latter is discussed in section 4 in analytic and numerical form using a Casas-Ibarra parameterisation and limits of small couplings. As lepton-flavour violation always occurs in this type of models, we compute the branching ratios for µ → eγ and similar processes in section 5. Note that the numerical results in section 3-5 have purely illustrative purpose for the specific topics. Section 6 contains our main numerical results from two random scans, one over the full parameter space, one in the coannihilation region of scalars and fermions. We impose all current experimental constraints, estimate the sensitivity of future experiments and discuss the possible impact of the LHC. Our conclusions are given in section 7.

Particle content and interactions in the model
In our model T1-2-A, the SM is extended by two scalars, a real singlet φ S and a complex doublet φ D , and two fermions, a Weyl singlet ψ S and a vector-like doublet ψ D , which can be decomposed into two chiral Weyl fermions ψ D 1 , ψ D 2 . The new (SM) particles are all assumed to be odd (even) under a discrete Z 2 symmetry to render dark matter stable and generate neutrino masses only at the loop level. Under the combined SU(2) L ⊗U(1) Y ⊗Z 2 symmetry group, the new particles carry the quantum numbers

JHEP10(2018)055
The pair of Weyl fermions are required for the cancellation of anomalies. For the doublets, the decompositions according to the physical charges Q = T 3 +Y /2 have also been indicated. It is evident that the model can be seen as a combination of the singlet-doublet scalar [34][35][36][37] and the singlet-doublet fermion dark matter model [34,35,[38][39][40], which have both well been studied separately. Our model allows in addition for the radiative generation of neutrino masses. The Lagrangian of the singlet-doublet scalar model involves in general five different couplings of the new scalars with the complex SM Higgs doublet H, which acquires a vacuum expectation value v = 246 GeV and thus breaks the electroweak symmetry. One of these couplings (A) links the singlet and doublet scalars with the SM Higgs in a Yukawa-like form. The Z 2 symmetry remains unbroken. We assume that the new scalars have positive mass squared and so do not acquire a vacuum expectation value that could induce trilinear scalar couplings. Note that quartic couplings in this case would only induce additional scalar conversions -as the Z 2 symmetry prevents mixing -that can also occur through other processes and are therefore not considered. The Lagrangian of the singlet-doublet fermion model contains a Majorana mass term for the singlet ψ S , a Dirac mass term for the doublet ψ D as well as two Yukawa terms connecting the singlet and doublet fermions with the SM Higgs. The presence of scalar and fermion singlets and doublets allows also to define couplings of the new particles to the SM leptons, is the SM lepton doublet of generation i. These new Yukawa couplings g ij , together with the interactions of eqs. (2.3) and (2.4), generate one-loop neutrino masses after electroweak symmetry breaking, while at tree level the neutrinos remain massless. A term g 3i e c i,R φ † D ψ D 1 with no effect on neutrino masses has been omitted. The presence of these non-zero neutrino masses implies lepton flavour violation. Inspection of eqs. (2.3)-(2.5) shows that the terms featuring A, y 1 , y 2 and g ij do not allow to determine the lepton numbers of the new particles consistently without violating lepton flavour. Only when either g ij or all mixing terms A, y 1 and y 2 vanish, lepton flavour symmetry can be restored. Neutrino sector g 11 , g 12 , g 13 , g 21 , g 22 , g 23 Table 1. Free parameters in model T1-2-A.
In the basis of real neutral scalars (φ S , φ R , φ I ), one can extract their mass matrix after electroweak symmetry breaking from the Lagrangian in eq.
(2.7) In the absence of new sources of CP-violation, φ I does not mix with either φ R or φ S . The mass matrix for the neutral fermions can similarly be derived from the Lagrangian in eq. (2.4) [35], so that Here, m y = 1 √ 2 yv with y = y 2 1 + y 2 2 , v is the SM Higgs vacuum expectation value, and tan θ = y 2 y 1 , so that y 1 = y cos θ and y 2 = y sin θ. Mixing of the neutral scalar and fermion particles is parameterised by the matrices U S and U F , which leads to the mass eigenstates with mass m ψ − = M D,F . In total, our model contains 17 free parameters, which are summarized in table 1. Diagonalization of the scalar mass matrix in eq. (2.7) leads to For a physical spectrum, this implies and thus for vanishing λ S , λ D etc. (2.14)

Dark matter relic density
Let us first review the fermion and scalar dark matter sectors of our model separately.
The singlet-doublet fermion model has been shown to be severely constrained by LUX and XENON1T. Only regions with small Yukawa coupling y < 0.1 are still allowed except for regions close to blind spots, and those are fine-tuned to at least 10%. The singlet-doublet scalar model was less constrained by LUX. In the limit λ = λ S = λ D with λ D = λ D = 0, regions of λ < 0 were, however, excluded by XENON1T, while λ > 0 remained viable, albeit again with a typical fine-tuning of ≤ 10% [35]. The correct relic density is reached in a thermal freeze-out through electroweak interactions and often through resonances or via coannihilation processes [34]. In both cases, the dark matter mass had to be in the few hundred GeV to few TeV range. Relic density and direct detection constraints are correlated through the same Higgs couplings, shown in the top row of figure 2. We have verified that we correctly reproduce the results in ref. [35] separately for fermion and scalar singlet-doublet dark matter. Our model allows for both fermion and scalar dark matter candidates as well as for fermion-scalar coannihilation processes, mediated by the lepton vertex shown in the bottom row of figure 2. This implies also that our dark sector can be leptophilic, if the Higgs couplings are small, and that new annihilation processes, shown in figure 3, can occur, which will in general relax the correlation of relic density and direct detection constraints JHEP10(2018)055 via the Higgs couplings. The annihilation cross sections of these processes scale with the relative velocity v of the thermal relic as respectively. Annihilations to charged leptons through the omitted couplings g 3i would therefore be velocity-suppressed. Conversion processes χ i χ j ↔ X k X l can also occur, so that the evolution of the dark matter particle is described by a set of coupled Boltzmann equations. We solve these with micrOMEGAs 4.0.3 [41] after implementation of the Lagrangians in eqs. (2.3)-(2.5) in LanHEP [42] and SARAH 4 [43] as a cross-check and to facilitate use of the spectrum generator SPheno 3.3.6 [44].
In figure 4, we demonstrate the impact of the couplings g ij among the new fermion and scalar sectors on the singlet scalar dark matter relic density as function of its mass. For each line the couplings g ij are fixed to the same value as stated on the plot. The other masses M D,S , M S,F and M D,F are each fixed to be a constant ratio of M S,S and larger by factors of 3, 3.1 and 2.5, respectively, so that coannihilations are unimportant. Mixings and (co-)annihilations in the scalar sector alone are further suppressed by small scalar couplings λ S etc., which are all set to values below 10 −5 , so that the relic density with g ij = 0 (dashed blue line) is seven orders of magnitude larger than the value observed by Planck (dotted black line). The lightest neutral fermion is then also stable, but annihilates into W -bosons thanks to its doublet component, with little impact on figure 4 as we have verified by changing g ij from 0 to 10 −5 . Increasing the scalar-fermion coupling to g ij = 2 (full red line) then allows to bring the singlet scalar relic density in agreement with the observation, whereas g ij = 1 (dot-dashed green line) is not quite sufficient. The direct detection cross section is independent of g ij , so that it can be decoupled from the relic density constraint. The parameters for this plot were chosen for illustrative purpose such that without the leptonic final states the relic is way too large and only the contributions from the new channels allow to reach a viable relic density. Yet it is important to mention that not  always all g ij of order one are needed to match the observation even when demanding a large contribution from leptonic final states. Next, we consider the effect of the couplings g ij on singlet fermion dark matter. Figure 5 shows its relic density as a function of the doublet scalar mass M D,S . The singlet and doublet fermion masses have been fixed at M S,F = 450 GeV and M D,F = 3 TeV, respectively, and the singlet scalar mass is 2.5 TeV, so that dark matter is always fermionic in the mass range shown. Since the Yukawa couplings y 1 = 0.06 and y 2 = 0.28 are small, it is also dominated by the singlet component. In the absence of scalar-fermion couplings g ij (dashed blue line), the relic density stays constant above M D,S = 600 GeV. The lightest neutral scalar is then also stable and annihilates into W -bosons thanks to its doublet component, with little impact on figure 5 as we have verified by changing all g ij from 0 to 10 −5 . Below M D,S = 600 GeV, the mass difference of the lightest physical scalar and fermion falls to a few tens of GeV and conversion processes χ i χ j ↔ X k X l are no longer Boltzmann-suppressed, which depletes the relic density to acceptable values when M D,S M S,F . When all the scalar-fermion couplings are set to g ij = 0.2 (dot-dashed green line) or even to g ij = 1 (full red line), the relic density falls monotonically with the doublet scalar mass, as coannihilations become more and more important. Even for large values of M D,S , the relic density is smaller than for g ij = 0 due to the presence of Their effect becomes smaller due to the scalar propagator suppression as M D,S increases. The direct detection cross section is, of course, again independent of g ij .
In figure 6, we analyse the scalar dark matter relic density in the mass plane M D,S -M S,S in the presence of light singlet or doublet fermions for a fixed, sizeable value of g ij = 0.75. The scalar couplings are either small (λ S = λ D = 10 −4 , A = 10 −4 GeV) or zero (λ D = λ D = 0), so that the dark matter particle is mostly a singlet when M S,S < M D,S and a doublet when M D,S < M S,S . The Yukawa couplings have been set to y 1 = 0.3 and y 2 = 0.2, respectively. In order to study the influence of coannihilation, the difference between fermion and scalar masses is reduced. In regions for M D,S smaller than M S,S the fermion singlet mass M S,F is chosen to have a relative difference indicated by the colored lines to M D,S , while the doublet fermion mass parameter M D,F stays at 5 TeV. For M S,S lighter than M D,S , the doublet fermion parameter M D,F is scaled accordingly, while M S,F is fixed at 5 TeV. For each contour the border indicates the region where the relic density matches the observations. Below or to the left of the contour, where the region is shaded, the relic density is below the observed limit. When the fermions are decoupled to M S,F = M D,F = 5 TeV (full black line), coannihilations cannot take place, and one recovers the result in figure 12 of ref. [35]. However, when the relative mass difference of singlet fermions to doublet scalars is in the few-percent range (coloured vertical lines), the allowed range in the relic density is extended beyond M D,S = 600 GeV to about 750 GeV. Conversely, when the relative mass difference of doublet fermions to singlet scalars is in the few-percent range (coloured horizontal lines), a new region of singlet scalar dark matter opens up due to scalar-fermion coannihilations, conversion processes and annihilations into SM leptons. For relative mass differences of 1-10% and these choices of parameters, the singlet scalar dark matter mass cannot exceed 900 and 250 GeV, respectively.

Neutrino masses
The combination of a singlet-doublet scalar and a singlet-doublet fermion sector in model T1-2-A allows for the radiative generation of neutrino masses through the box diagram depicted in figure 1 (upper right) and thus for a natural explanation of the small relative size of neutrino masses with respect to other fermion masses. Radiative neutrino mass models JHEP10(2018)055 have been reviewed extensively in ref. [25] including both radiative Dirac and Majorana mass schemes. The latter can be classified according to the loop-order realization of the Weinberg operator. Here, we focus on the one-loop realization of the d = 5 operator [22]. After electroweak symmetry breaking, Majorana neutrino masses in our model are generated at one loop by the diagram shown in figure 7. As explained in table 37 of ref. [21], this includes the possibility of neutrino mass generation through diagrams of topology T1-1-A, T1-3-A and T3-B. The neutrino mass matrix is then given by which can be written as M ν = g T M g with the scalar-fermion coupling matrix g = g 11 g 12 g 13 the elements of the symmetric matrix M 3) and the mass function In the limit of vanishing λ S , λ D etc. and small values of A, y 1 , y 2 1, the matrix elements of M can be expanded and expressed as

JHEP10(2018)055
This demonstrates that the generation of non-zero neutrino masses requires non-vanishing values of g ij , A, y 1 and/or y 2 . In the opposite limit of vanishing A, y 1 , y 2 etc. and small values of λ D , one finds i.e. the neutrino masses are proportional to the doublet scalar mass splitting. The diagonalization of the neutrino mass matrix D ν = U T ν M ν U ν = (0, m ν 2 , m ν 3 ) with the PMNS matrix U ν leads to two non-zero Majorana neutrino masses, while the third mass is always zero. The observed neutrino mass differences [45] then translate directly to absolute neutrino masses. A third non-zero neutrino mass could in principle be accommodated, but would require at least an additional scalar or fermion singlet. Assuming normal mass hierarchy, the experimental constraints on the neutrino masses and mixing angles translate directly into constraints on the couplings [46]  depends on a single parameter ϕ. For definiteness, we take all parameters entering eq. (4.7) to be real, i.e. the Dirac and Majorana phases in U ν and the phase associated with R are assumed to be zero. In our numerical studies we assume normal hierarchy when imposing neutrino mass and mixing constraints. From eq. (4.1) it is clear that in the absence of all scalar-fermion couplings g ij , neutrino masses cannot be generated. The neutrino masses depend, however, also on the dark mass spectrum and, through the fermion and scalar mixing matrices U F and U S , on the Yukawa couplings y 1 and y 2 , the singlet-doublet scalar coupling A and, to a lesser extent, on the exclusively singlet or doublet scalar couplings λ S , λ D etc. This can also be seen from figure 2 of ref. [21] before electroweak symmetry breaking, which involves all four dark mass parameters as well as the couplings g 2 ij , y 1 or y 2 , and A, but not λ S , λ D etc. In the following numerical studies of the total neutrino mass sum, we therefore fix the latter to values of O(10 −1 ) or smaller.
In figure 8, we demonstrate the influence of the scalar-fermion couplings g ij on the sum of neutrino masses as a function of the scalar singlet mass regarding the diagram shown in figure 8. The heavier particles are suppressed in the loop, so that the singlet scalar-doublet fermion contribution related to g 1i dominates, at least to some extent. As the singlet scalar mass (and with it all other masses) increases, the neutrino masses decrease moderately as expected from the propagator suppression in the loop. More importantly, as the scalar-fermion couplings g ij , and in particular the singlet scalar-doublet fermion couplings g 1i , decrease from values of 1 (dashed blue line) to 10 −2 (dot-dashed green line) and 10 −4 (full red line), the neutrino mass sum drops over eight orders of magnitude in agreement with the quadratic scaling of M ν,ij with g ij in eq. (4.1). In this scenario, the viable value of i m ν i ≈ 5×10 −11 GeV [45] is thus reached only for very small values of g ij . This is, however, not always the case, as we demonstrate in figure 9. Here, the neutrino mass sum is shown as a function of the singlet fermion mass M S,F , while the other masses are scaled to it via M S,S = 2.5 M S,F , M D,S = 3.5 M S,F and M D,F = 1.5 M S,F . Again all g ij are set to a fixed value for each line as indicated in the plot. The doublet scalar mass is substantially larger than in the previous figure, and it mixes strongly (A = 10 GeV) with the singlet scalar, while the Yukawa couplings are comparable (y 1 = 4 · 10 −2 , y 2 = 2 · 10 −2 ). Therefore, the neutral scalar components are no longer mostly singlet-or doublet-like and the contributions of the g 1i and g 2i Lagrangian terms no longer strongly separated, so that the dominance of the singlet fermion and the doublet scalar in the loop is less pronounced. In total, this leads to neutrino mass sums that are at least an order of magnitude smaller and fall more steeply for small M S,F . The striking feature in figure 9 is the cancellation  similar size, they can be larger to be in the minimum region than when this is true for only one of them. As the doublet fermion mass decreases and approaches the scalar singlet mass (bottom right), the Yukawa couplings also drop in order for the neutrino masses to remain around the smaller values. Interestingly, they can not be too small either once the doublet fermion mass drops below the singlet fermion mass (beyond M S,S /M S,F = 0.3). The doublet fermion then has to mix more with the singlet fermion to reduce the neutrino mass to the viable value. This figure along with eq. 4.5 indicates that for small y 1,2 the values for g ij can be larger while still reaching desirable values for the neutrino masses. The absolute scale of the sum of neutrino masses depends not only on the doublet fermion mass M D,F but also on the singlet fermion and doublet scalar mass parameters M S,F and M D,S . Changing these masses allows to change the scale and therefore the region of suppression for the neutrino masses while mostly keeping a similar behaviour regarding y 1 and y 2 as shown in the figure. through the bubble (top) or triangle (bottom) diagrams depicted in figure 11, where always both new scalars and fermions must run in the loops. Neglecting the external lepton masses, the branching ratio can be expressed as

JHEP10(2018)055
and similarly for τ → eγ and τ → µγ. It is obvious that in the limit of vanishing scalarfermion couplings g ij → 0, not only the neutrino masses, but also the lepton-flavour violating processes disappear. While the neutrino mass matrix in eq. (4.1) always depends on both the dark scalar and fermion mixing matrices U S and U F , the charged lepton processes always require one charged dark particle in the loop, so that they depend at the amplitude level only on one of the neutral mixing matrices. The same diagrams as in figure 11 also contribute to conversion processes, when the photon is taken off-shell and couples to the heavy nucleus. The flavour-violating leptonic decay processes µ → 3e etc. are mediated by the diagrams shown in figure 12. They involve bubble, triangle and box one-loop topologies and thus at the amplitude level two (bubbles and triangles) or four (boxes) powers of scalar- Figure 12.  [55] µ − Au → e − Au 7.0×10 −13 [56] - fermion couplings g ij and mixing matrices. Note that the omitted couplings g 3i would not induce qualitatively new contributions to these lepton-flavour violating processes. Current and future limits on the branching ratios for the processes described above are listed in table 2. As one can see, the process µ → eγ has the smallest branching ratio and thus the potential to set most stringent limits on new particle masses and couplings. Of course, this strongly depends on other model parameters as well. When using the limits from the table to constrain the parameter space, it is therefore advisable to use all available constraints and to not only consider the process µ → eγ.
We show in figure 13 the dependence of the branching ratio for this process on the singlet scalar mass M S,S . All other parameters have been set as in figure 8 to allow for an easy comparison with the corresponding neutrino masses. In particular, we show again three curves for all scalar-fermion couplings g ij of 1 (dashed blue curve), 10 −2 (dot-dashed green curve) and 10 −4 (full red curve). As expected, the branching ratio falls with the singlet scalar mass in the loop, but much more directly with the scalar-fermion couplings g ij . Current and future experimental limits can be evaded already with values of 10 −2 , whereas the neutrino masses required even smaller values of about 10 −4 for this specific set of couplings. Other parameter points can feature larger values of g ij while still evading the lepton flavour limits especially when not all g ij are simultaneously large. The previous sections 3 and 4 indicate that other model parameters such as y i or λ S provide further handles on experimental constraints and can compensate g ij while still matching the neutrino mass observations. Furthermore, they show that dark matter observables might need larger values for g ij which are more constrained by the lepton flavour limits. In the following section we will show that it is still possible to obtain parameter points which obey all these counteracting limits.

JHEP10(2018)055 6 Numerical results
In this section, we present our main numerical results. We focus on the case of scalar dark matter and perform two random numerical scans, the first over the full parameter space, the second in a region where coannihilation processes of singlet scalar and doublet fermion processes become important. We impose all available experimental constraints, in particular on the mass M H = 125 GeV and SM couplings of the Higgs boson [45], the observed dark matter relic density Ω obs h 2 = 0.1186 ± 0.0031 [14], and the neutrino masses and mixing angles [7]. The neutrino constraints are imposed using eq. (4.7). The squared mass differences are used in D ν and the mixing angles in U ν . It is assumed that they follow a Gaussian distribution around the central value within the observed errors under the assumption of normal hierarchy. The parameter ϕ is chosen randomly. The expression in eq. (4.7) then allows to obtain values for g ij which match observations. Limits on the direct detection cross section and lepton-flavour violating processes (cf. table 2) are shown explicitly for the first scan and imposed on the second scan. In both cases, we have obtained O(10 4 ) viable points. Possible LHC constraints are discussed at the end of this section.
In figure 14, we plot the spin-independent direct detection cross section σ SI as a function of the physical mass of scalar dark matter, which can be a mixture of the singlet and doublet. In particular, their mixing can be large and lead to physical masses of a few GeV despite the fact that the scanned mass parameters lie above 200 GeV. When the scalar dark matter does not couple to fermions (blue crosses), it is well known that the narrow relic density constraint is strongly correlated with the direct detection cross section. This leads to a narrow band below and on the Higgs resonance at M DM M H /2, which becomes somewhat wider above. For constant Ωh 2 , the Higgs coupling and direct detection cross section get smaller as the DM mass increases towards the Higgs resonance. On the resonance, annihilation is very efficient and the couplings and direct detection cross section must be very small. As expected for singlet-doublet scalar dark matter alone [34,35], most of the parameter points are now excluded by PandaX [59,60] (dashed curve) and XENON1T [61] (full curve). Furthermore, they obviously cannot explain the neutrino masses. The coupling to the fermion sector opens up new regions of parameter space (red squares) allowed by all experimental constraints including neutrino experiments that cannot even be probed by XENONnT [62], since the direct detection cross section is independent of the couplings g ij at tree-level. At one loop, fermion triangles can in principle contribute to the scalar direct dection cross section, but these contributions are mass suppressed, since we chose scalar and Yukawa couplings to be of similar size. The correct relic density is reached through annihilation processes into leptons and/or scalar-fermion coannihilations. As figure 15 shows, most of the new points are, however, excluded by the leptonflavour branching ratio BR(µ → eγ). Smaller direct detection cross sections imply small couplings to the Higgs and usually demand larger g ij in order to obtain the correct relic density. The smallest values of the Higgs couplings and hence σ SI are featured by mostly singlet-like dark matter which is mainly impacted by g 1i leaving g 2i rather unconstrained by relic density obersvations. Nevertheless, some of the points fulfill in fact all current experimental constraints including all lepton flavour constraints. Interestingly, they lead to direct detection cross sections that will soon be tested by XENONnT. The points evading all constraints are either close to the Higgs resonance, reducing the limitations on g ij regarding dark matter observables or feature small enough g ij to avoid lepton-flavour bounds which are compensated by intermediate dark matter Higgs couplings like λ S . limit the mixing in the scalar and fermion sectors by reducing the scan ranges of |λ S |, A, |y 1 | and y 2 to values below π · 10 −4 . This also enhances the annihilation into lepton final states.

Coannihilation region
After imposing all experimental constraints, including those on the direct detection cross section and lepton-flavour violating processes, we obtain the scalar-fermion couplings g ij shown in figure 16. Since A, y 1 and y 2 are now all small, we can obtain viable neutrino masses for sizeable values of g ij . As figure 16 shows, at least one of these couplings must be large, but they cannot be both large at the same time, which reflects the two rather different neutrino mass differences. Due to the weaker limits on lepton-flavour violation for processes involving the τ lepton, the values of g 13 are less restricted than those of g 11 and g 12 . As the singlet scalar mass M S,S and with it the doublet fermion mass M D,F increase, so must the couplings g ij to compensate for the propagator suppression in the neutrino mass loops. Conversely, as λ D and with it the doublet scalar mass splitting and the neutrino masses increase (cf. eq. 4.6), the corresponding scalar-fermion couplings g 2i must decrease for the neutrino masses to remain in the viable range (cf. figure 17). Since dark matter is chosen to be mostly singlet-scalar-like, the impact of dark matter constraints and lepton flavour violation bounds is much stronger for g 1i than for g 2i . The couplings g 2i are mostly associated with the scalar doublet and are mostly limited by neutrino mass limits as indicated by figure 17, which shows the need for compensating small g 21 by a larger λ D . The possibility of co-annihilation between scalars and fermions allows to obtain viable relic densities even though g ij can be relatively small. Figure 16. Viable scalar-fermion couplings in the plane |g 11 |-|g 12 | in the coannihilation region (colour code for |g 13 |). All points fulfill relic density, direct detection, neutrino mass and mixing as well as lepton-flavour violation constraints.

JHEP10(2018)055
A similar plot to the one in figure 16 for g 1i shows the plane of the Yukawa couplings |y 1 |-y 2 in figure 18. Again, one of these couplings, but not both must be relatively large, in particular when the singlet fermion is decoupled (blue points). However, when it becomes light (green points), mixes with the doublet fermion and contributes to coannihilation, smaller values for both Yukawa couplings also become viable. The direct detection cross section is governed by the coupling λ S of the scalar singlet to the Higgs boson. It ranges from 10 −27 pb for λ S = 10 −9 to 10 −14 pb for λ S = 10 −4 and is thus beyond the reach even of XENONnT. The situation is therefore similar to the one in the previous section, where many new viable models had very small direct detection cross sections and had to be constrained by lepton-flavour violating processes. We therefore present in figure 19 the branching ratios BR(µ → eγ), BR(τ → eγ) and BR(µ → 3e). Current and future experimental limits are indicated by full and dashed black lines. Since we impose the current limits on our scan, no excluded models are found. A substantial fraction of the viable models will be tested by future muon decay experiments in either of the two channels, while the sensitivity of future tau decay experiments remains limited and would have to be improved by at least two orders of magnitude to completely probe the upper one of the two viable parameter regions, where g 11 stays relatively constant. Figure 17. Viable scalar-fermion coupling |g 21 | vs. scalar coupling λ D in the coannihilation region. All points fulfill relic density, direct detection, neutrino mass and mixing as well as lepton-flavour violation constraints.

LHC constraints
In our first random scan over the full parameter space of scalar dark matter, the latter can be a mixture of singlet and doublet. Various collider limits on singlet and doublet scalar dark matter, in particular from Higgs invisible decays at the LHC and the charged scalar partners at LEP and the LHC, have been discussed in the past [63].
If dark matter is dominated by the singlet component, as it is also the case in our second random scan over the scalar-fermion coannihilation region, it couples neither to the photon nor to the weak gauge bosons, but only to the Higgs boson through the coupling λ S . Constraints from the LHC thus currently come only from the invisble decay width of the Higgs boson in a mass region below 62.5 GeV. The upper limits lie currently at 67% for associated ZH production in ATLAS [64] and 24% for a combination of different production channels in CMS [65]. As our figure 15 and figure 8b in ref. [65] show, models with such large couplings are already ruled out by direct detection and/or BR(µ → eγ). ref.
[67] lie at 50-100 GeV, so that the LHC would exclude only a few models beyond the Higgs resonance region. These limits apply more directly to the case of fermion dark matter, which we have not discussed.
Alternatively, heavier charged and/or neutral scalars can be produced at the LHC through weak gauge bosons. They decay subsequently into dark matter and W -or Zbosons, leading to two-or multi-lepton signals with missing transverse energy (see figure 20 left) [68]. These signals have been analysed at the LHC mostly in the context of charged scalar leptons appearing in supersymmetric models. Scalar neutrinos are so far unconstrained by the LHC. Although the charged sleptons do not translate directly into our model, since they decay into fermionic neutralino dark matter in supersymmetry, we can again assume spin to be of minor importance and our charged scalars to decay into the lightest neutral scalars and weak bosons with branching ratios of one, which leads to the strongest constraints. Similar assumptions are also used in the slepton analyses of AT-LAS [69] and CMS [70], which leads to lower slepton mass limits of 520 GeV in figure 6b of ref.
[69] and 440 GeV in figure 3 of ref. [70]. The corresponding mass limits for dark matter then range from 50-280 GeV and 40-220 GeV, respectively, which would in principle exclude some of the models beyond the Higgs resonance region in figure 15. The limits in our model will, however, be considerable weaker since one has to take into account in addition the leptonic branching fractions of the W -and Z-bosons of 21% and 7% for electrons and muons, respectively. If the charged scalars are heavier than 440-520 GeV or if they have other (e.g. cascade) decays, the LHC limits of course no longer apply at all. JHEP10(2018)055 Figure 19. Predicted branching ratios in the coannihilation region for the most sensitive leptonflavour violating processes BR(µ → eγ), BR(τ → eγ) and BR(µ → 3e) (colour) with current (full black lines) and future (dashed black lines) exclusion limits. All points fulfill relic density, direct detection, neutrino mass and mixing as well as current lepton-flavour violation constraints. Figure 20. Typical diagrams for the production of heavy scalars (left) and fermions (right) decaying into two-or multi-lepton final states and missing transverse energy, carried away by scalar dark matter X 1 and neutrinos.
Doublet fermions, which coannihilate in our second random scan with singlet dark matter, can be constrained in a similar way, in particular from searches for higgsino-like charginos and neutralinos (see figure 20 right) [28]. Their masses have been constrained by LEP and ATLAS in figure 10 of ref.
[71] to be at least 95-145 GeV and by CMS in figure 8 of ref.
[72] to be at least 100-170 GeV. The fermionic neutralino dark matter mass then has to be 95-140 GeV and 100-150 GeV, respectively. This would not affect many of our viable models in figure 15 beyond the Higgs resonance region. However, the limits in our model will here be stronger, since one does not have into account the leptonic branching fractions JHEP10(2018)055 of the W and Z-bosons. For singlet-doublet fermion dark matter with scalar singlets, the charged fermions have been shown to require masses of at least 510 GeV [28].

Conclusion
In conclusion, we have analysed in this paper a combination of the singlet-doublet scalar and singlet-doublet fermion dark matter models, which had so far only been analysed separately. Their combination allowed for the radiative generation of neutrino masses, but also admitted lepton-flavour violating processes.
After discussing the analytic structure and main parameter dependencies of the model and the implications for the dark matter relic density, neutrino masses and lepton flavour violation, we performed two random scans of the parameter space, focusing on the case of scalar dark matter.
In the first scan over the full parameter space, we imposed contraints from the Higgs mass, relic density and neutrino masses and mixings using the Casas-Ibarra parameterisation. We found that the scalar dark matter could be a mixture of singlets and doublets and in particular that the scalar-fermion couplings opened up large new regions of parameter space, mostly with direct detection cross sections that will escape experimental verification way beyond XENONnT. Many of these models were instead shown to be excluced by lepton-flavour violation constraints. The remaining viable models will soon be probed by XENONnT.
In the second scan, we focused on singlet scalar dark matter coannihilating with mostly doublet fermions. In this case we also imposed constraints from direct detection and leptonflavour violation experiments. We found that at least one of the scalar-fermion couplings and Yukawa couplings had to be large or not too small, respectively, leading to two distinct regions in parameter space. Many of these models will soon be tested by experiments on µ → eγ or µ → 3e, while those on τ → eγ would require an increase of at least two more orders of magnitude beyond current planning to allow for the complete testing of at least one of the two viable regions in parameter space.
LHC constraints from invisible Higgs decays were shown to add no further constraints on the models with dark matter masses below the Higgs resonance, while monojet, monoboson and in particular dilepton searches have in principle the potential to reach into the region beyond it up to 280 GeV. These limits depend, however, crucially on the heavier and in particular charged particle mass spectrum and decay modes. A full analysis of the model would require detailed information on acceptances and efficiencies of the LHC experiments and is beyond the scope of this work.