1 Introduction

The existence of dark matter (DM) in the Universe is highly motivated by many astrophysical and cosmological observations [14]. However, its nature remains a mystery. One of the best motivated candidates for a DM particle is a weakly interacting massive particle (WIMP) [5]. At the Large Hadron Collider (LHC), one can search for DM particles (\(\chi \)) that are pair produced in \(pp\) collisions. These studies are sensitive to low DM masses (\(m_{\chi }\le \) 10 GeV), and therefore provide information complementary to direct DM searches, which are most sensitive to larger DM masses  [69].

If the particles that mediate the interactions between DM and Standard Model (SM) particles are too heavy to be produced directly in the experiment, their interactions can be described by contact operators in the framework of an effective field theory [1012]. For each operator considered, the reach is expressed in terms of the effective mass scale of the interaction, \(M_*\), and of the \(\chi \)–nucleon cross-section, \(\sigma _{\chi -\mathrm{N}}\), as a function of \(m_\chi \).

Since DM particles do not interact in the detector, the main signature of DM pair production at colliders is large missing transverse momentum. Initial-state radiation (ISR) of jets, photons, \(Z\), or \(W\) bosons, was used to tag DM pair production at colliders in several searches at the Tevatron [13] and the LHC [1422].

A new search for DM pair production in association with one \(b\)-quark or a pair of heavy quarks (\(b\) or \(t\)) was proposed in Ref. [23]. The dominant Feynman diagrams for these processes are shown in Fig. 1. To search for these processes, dedicated selections are defined to reconstruct the various production and decay modes of these heavy-quark final states. For final states containing a semileptonic decay of a top quark, the results of the search for a supersymmetric partner of the top quark are used [24].

Fig. 1
figure 1

Dominant Feynman diagrams for DM production in conjunction with a a single \(b\)-quark and b a heavy quark (bottom or top) pair using an effective field theory approach

Fig. 2
figure 2

Example of DM production in the \(b\)-FDM model

The analysis presented in this article is particularly sensitive to effective scalar interactions between DM and quarks described by the operator [12]

$$\begin{aligned} \mathcal{O}_\text {scalar}= \sum _q \frac{m_q}{M_*^N}\bar{q}q\bar{\chi }\chi , \end{aligned}$$
(1)

where \(N=3\) for Dirac DM (D1 operator) and \(N=2\) for complex scalar DM (C1 operator). The quark and DM fields are denoted by \(q\) and \(\chi \), respectively. The scalar operators are normalized by \(m_q\), which mitigates contributions to flavour-changing processes, strongly constrained by flavour physics observables [25, 26], through the framework of minimal flavour violation (MFV). The dependence on the quark mass makes final states with bottom and top quarks the most sensitive to these operators.

This search is also sensitive to tensor couplings between DM and quarks. The tensor operator (D9), which describes a magnetic moment coupling, is parameterized as [12]:

$$\begin{aligned} \mathcal{O}_\text {tensor} = \sum _q \frac{1}{M_*^{2}}\bar{\chi }\sigma ^{\mu \nu }\chi \bar{q}\sigma _{\mu \nu }q. \end{aligned}$$
(2)

MFV suggests that the D9 operator should have a mass dependence from Yukawa couplings although canonically this is not parametrised as such.

The results are also interpreted in light of a bottom-Flavoured Dark Matter model (\(b\)-FDM) [27]. The \(b\)-FDM model was proposed to explain the excess of gamma rays from the galactic centre, recently observed by the Fermi Gamma-ray Space Telescope, and interpreted as a signal for DM annihilation [28]. This analysis of the data recorded by the Fermi-LAT collaboration favours DM with a mass of approximately 35 GeV annihilating into \(b\)-quarks via a coloured mediator. In this model, a new scalar field, \(\phi \), mediates the interactions between DM and quarks as shown in Fig. 2. DM is assumed to be a Dirac fermion that couples to right-handed, down-type quarks. The lightest DM particle, which constitutes cosmic DM, preferentially couples to \(b\)-quarks. The collider signature of this model is \(b\)-quarks produced in association with missing transverse momentum. This analysis sets constraints on the mass of the mediator and DM particle in the framework of the \(b\)-FDM model.

Table 1 Selections for signal regions 1–4. Variables \(p_\mathrm {T}^{j_i}\) (\(p_\mathrm {T}^{b_i}\)) represent the transverse momentum of the \(i\)th jet (\(b\)-tagged jet). The asymmetric transverse mass \(am_\mathrm {T2}\) [2931], \(topness\) [32], \(m_{jjj}\) and Razor \(R\) [33] are used to reject the abundant top quark background

2 Detector description and physics objects

The ATLAS detector [34] at the LHC covers the pseudorapidityFootnote 1 range of \(| \eta | <4.9\) and is hermetic in azimuth \(\phi \). It consists of an inner tracking detector surrounded by a superconducting solenoid, electromagnetic and hadronic calorimeters, and an external muon spectrometer incorporating large superconducting toroidal magnets. A three-level trigger system is used to select events for subsequent offline analysis. The data set used in this analysis consists of \(20.3 \mathrm {~fb}^{-1}\) of \(pp\) collision data recorded at a centre-of-mass energy of \(\sqrt{s} = 8\) TeV with stable beam conditions [35] during the 2012 LHC run. All subsystems listed above were required to be operational.

This analysis requires the reconstruction of muons, electrons, jets, and missing transverse momentum. Muon candidates are identified from tracks that are well reconstructed inside both the inner detector and the muon spectrometer [36]. To reject cosmic-ray muons, muon candidates are required to be consistent with production at the primary vertex, defined as the vertex with the highest \(\Sigma (p_\mathrm {T}^{\mathrm {track}} )^2\), where \(p_\mathrm {T}^{\mathrm {track}}\) refers to the transverse momentum of each track.

Electrons are identified as tracks that are matched to a well-reconstructed cluster in the electromagnetic calorimeter. Electron candidates must satisfy the tight electron shower shape and track selection criteria of Ref. [37]. Both electrons and muons are required to have transverse momenta \(p_\mathrm {T}\ > 20\) GeV and \(|\eta |<2.5\). Potential ambiguities between overlapping candidate objects are resolved based on their angular separation. If an electron candidate and a jet overlap within \(\Delta R< 0.2\), then the object is considered to be an electron and the jet is discarded. If an electron candidate and any jet overlap within \(0.2 < \Delta R < 0.4\), or if an electron candidate and a b-tagged jet overlap within \(\Delta R< 0.2\) of each other, then the electron is discarded and the jet is retained.

Photon candidates must satisfy the tight quality criteria and \(|\eta | < 2.37\) [38].

Jet candidates are reconstructed using the anti-\(k_t\) clustering algorithm [39] with a radius parameter of 0.4. The inputs to this algorithm are three-dimensional topological clusters [40]. The four-momentum of the jet is defined as the vector sum of the four-momenta of the topological clusters, assuming that each cluster originates from a particle defined to be massless and to come from the interaction point.

To calibrate the reconstructed energy, jets are corrected for the effects of calorimeter response and inhomogeneities using energy- and \(\eta \)-dependent calibration factors based on simulation and validated with extensive test-beam and collision-data studies [40]. In the simulation, this procedure calibrates the jet energies to those of the corresponding jets constructed from stable simulated particles. In-situ measurements are used to further correct the data to match the energy scale in simulated events. Effects due to additional \(pp\) interactions in the same and preceding bunch crossings (pile-up effects) are corrected [41]. Only jets with \(p_\mathrm {T}>20 (25)\) GeV and \(|\eta |<4.5 (2.5)\) are considered in this analysis for final states involving \(b\) (\(t\)) quarks.

Jets containing particles from the hadronisation of a \(b\)-quark (\(b\)-jets) are tagged using a multivariate algorithm [42, 43]. The \(b\)-tagging algorithm combines the measurement of several quantities distinguishing heavy quarks from light quarks based on their longer lifetime and heavier mass. These quantities include the distance of closest approach of tracks in the jet to the primary event vertex, the number and position of secondary vertices formed by tracks within the jet, as well as the invariant mass associated with such vertices. The algorithm is trained on Monte Carlo (MC) simulations and its performance is calibrated using data. To optimize the sensitivity of this analysis, a requirement on the output of the \(b\)-tagging algorithm which provides a 60 % (70 %) \(b\)-jet efficiency operating point is used in signal regions (SR) 1 and 2 (3 and 4) defined below. The corresponding misidentification probability is 15 % (20 %) for \(c\)-jets, and less than 1 % for light-quark jets. The aforementioned \(b\)-tagging efficiencies and misidentification probabilities were derived in a simulated \(t\bar{t}\) sample with jet transverse momenta of \(p_\mathrm {T}>20\) GeV and \(|\eta |<2.5\).

The missing transverse momentum, with magnitude \(E_\mathrm {T}^{\mathrm {miss}}\), is defined as the negative vector sum of the transverse momenta of jets, muons, electrons, photons, and topological clusters not assigned to any reconstructed objects [44].

Table 2 Expected background and signal yields for \(m_\chi =10\) GeV compared with observed yields in data for the various signal regions. For the \(b\)-FDM model, \(m_\phi \) is 600 GeV. The row labeled “total expected background” shows the sum of all background components. The quoted uncertainties include all statistical and systematic effects added in quadrature. The effective mass scale, \(M_*\), is set to be 100/40/600 GeV for the D1/C1/D9 operators, approximately corresponding to the expected limit. The probabilities of the background-only hypothesis, \(p\) values, are also given. The last two lines show the observed and expected 95 % CL upper limits on the number of beyond-the-SM events

3 Event selection

Candidate signal events containing at least one high-\(p_\mathrm {T}\) jet and large \(E_\mathrm {T}^{\mathrm {miss}}\) are assigned to one of four orthogonal signal regions. The first two signal regions focus on events with DM produced in conjunction with one (SR1) or two (SR2) \(b\)-quarks in the final state. SR3 and SR4 target events in which DM is produced in conjunction with a \(t\overline{t}\) pair, where either both top quarks decay hadronically (SR3) or one top quark decays hadronically and the other semileptonically (SR4). SR4 was developed for a top squark search by the ATLAS Collaboration and coincides with the “tNbC_mix” signal region described in Ref. [24]. The four signal regions provide the complementary information needed in case of observation of a signal.

Events assigned to SR1 and SR2 are required to pass a calorimeter-based \(E_\mathrm {T}^{\mathrm {miss}}\) trigger with a threshold of 80 GeV. To enrich the sample in \(pp \rightarrow \chi \bar{\chi }+b(\bar{b})\), events are required to have a low jet multiplicity (\(n_{\text {jets}}<\)5), \(E_\mathrm {T}^{\mathrm {miss}}\) \(>\) 300 GeV, and the most energetic \(b\)-tagged jet must have a \(p_\mathrm {T}\) \(>\) 100 GeV. The azimuthal separation between the directions of the jets and the missing transverse momentum is required to be more than 1.0 radian. Events with at least one identified muon or electron are discarded to reject leptonic decays of \(W\) and \(Z\) bosons. Events satisfying these selection criteria are assigned to SR1 provided that the jet multiplicity does not exceed two. Events are assigned to SR2 when at least three jets are reconstructed in the event and the second most energetic jet has \(p_\mathrm {T}> 100\) GeV. If there is a second \(b\)-tagged jet it has to satisfy \(p_\mathrm {T}> 60\) GeV.

Events assigned to SR3 are required to pass triggers specifically designed to select hadronic decays of top quark pairs. Such triggers require either five jets with \(p_\mathrm {T}\ge \) 55 GeV each or four jets with \(p_\mathrm {T}\ge \) 45 GeV, of which one is tagged as a \(b\)-jet. To select \(pp \rightarrow \chi \bar{\chi }+t\bar{t} \) events, at least five reconstructed jets are required, of which at least two are \(b\)-tagged, and \(E_\mathrm {T}^{\mathrm {miss}}> 200\) GeV. Furthermore, the azimuthal separation between the most energetic \(b\)-jet and the missing transverse momentum is required to be at least 1.6 radians. To reduce \(W/Z\) leptonic decays and leptonic top quark decays, events with at least one identified muon or electron are discarded. To maximize the rejection of the abundant \(t\bar{t}\) background, the Razor variable \(R\) [33] is used. This variable utilizes both transverse and longitudinal information about the event to fully exploit the kinematics of the decay. To separate signal and background, \(R >0.75\) is required.

Fig. 3
figure 3

Comparison between data and expected SM background. a, b \(E_\mathrm {T}^{\mathrm {miss}}\) variable for SR1 and SR2 and for an example signal with the operator D9. c \(R\) variable for SR3 excluding the selection on \(R\) and for an example signal with the operator D1. d \(E_\mathrm {T}^{\mathrm {miss}}\) variable for SR4 excluding the selection on \(E_\mathrm {T}^{\mathrm {miss}}\)and for an example signal with the operator D1. Other backgrounds are composed of diboson and multijet production. The expected signal for \(\chi \bar{\chi } + b(\bar{b})\) (SR1, 2) and for \(\chi \bar{\chi } + t\bar{t}\) (SR3, 4) production for \(m_\chi =\) 10 GeV is given by the red line assuming \(M_*= 100/40/600\) GeV for the D1/C1/D9 operators, respectively. The final selection requirements are indicated by an arrow. The error bars represent the statistical uncertainty. The dashed area shows the systematic uncertainty on the background estimation. Events with values exceeding the range presented are included in the highest bin

To enrich the sample in \(pp \rightarrow \chi \bar{\chi }+t\bar{t}\) with one semileptonic decay of the \(t\) quark, events assigned to SR4 use single-lepton or \(E_\mathrm {T}^{\mathrm {miss}}\) triggers, and require exactly one isolated lepton (electron or muon) with \(p_\mathrm {T}> 25\) GeV, at least four high-\(p_\mathrm {T}\) jets, where one jet is \(b\)-tagged with \(p_\mathrm {T}> 60\) GeV. Events with \(E_\mathrm {T}^{\mathrm {miss}}>270\) GeV are selected when the transverse massFootnote 2 formed by the lepton and \(E_\mathrm {T}^{\mathrm {miss}}\), \(m_\mathrm {T}(\ell ,E_\mathrm {T}^{\mathrm {miss}})\), exceeds 130 GeV and \(E_\mathrm {T}^{\mathrm {miss}}/ \sqrt{H_\text {T}^{4j}} > 9~\sqrt{\text {GeV}}\), with \(H_\text {T}^{4j} = \sum _{i=1}^4 p_\mathrm {T}(\text {jet}_i)\) and where the jets are ordered by decreasing \(p_\mathrm {T}\). The azimuthal angle between the missing transverse momentum and the two most energetic jets is required to be greater than 0.6 radians.

Special variables, such as the asymmetric transverse mass \(am_\mathrm {T2}\) [2931] and the \(topness\) variable [32], are used to reject the dileptonic \(t\bar{t}\) component of the background. Details can be found in Ref. [24]. The diboson background is suppressed by a requirement on the three-jet invariant mass (\(m_{jjj}<360\) GeV) [24]. A \(\tau \) veto rejects \(t\bar{t}\) events with hadronically decaying \(\tau \) leptons in the final state. Additional selection criteria [24] on the angles between the lepton and the various jets are imposed to further reduce the \(t\bar{t}\) background. Table 1 provides an overview of the selections applied in all four signal regions.

The product of the detector acceptance \({A}\) and the reconstruction efficiency \(\epsilon \) for the selections described above varies between 0.1 and 8 % depending on the signal region, operator, and specific channel considered. SR1 and SR2 have the highest efficiencies (\({A} \times \epsilon >2\,\%\)) for the D9 operator, while SR3 and SR4 are most efficient for the D1 and C1 operators (\({A} \times \epsilon >1\,\%\)).

The dominant background for SR1 and SR2 is due to \(Z\rightarrow \nu \overline{\nu }\) events produced in conjunction with one or more jets. This irreducible background is estimated from data using two control regions (CRs). The first CR exploits \(Z+\)jets events with \(Z\rightarrow \mu ^+\mu ^-\), while the second uses \(\gamma +\)jets events for which the production at high transverse momentum (\(p_\mathrm {T}^{\gamma }>M_Z\)) mimics that of \(Z+\)jets [45]. The \(\gamma +\)jets control region substantially increases the number of events at large missing transverse momentum. The transverse momentum of the dimuon pair or photon is added vectorially to the \(E_\mathrm {T}^{\mathrm {miss}}\) of the event to simulate the \(Z\rightarrow \nu \bar{\nu }\) background. Corrections to compensate for the differences in efficiency and acceptance between the \(Z(\nu \overline{\nu })+\)jets and \(Z(\mu ^+\mu ^-)+\)jets or \(\gamma +\)jets are derived from data using control regions without \(b\)-tagged jets before applying any requirements on the missing transverse momentum. Remaining kinematic selections correspond to the ones described in Table 1. A muon control region is chosen because the energy loss of muons in the detector is comparatively small. The systematic uncertainties introduced by this data-driven procedure on the \(Z(\nu \bar{\nu })\)+jets background are approximately \(10\) %, mainly from the flavour composition of background processes, kinematic differences between the control and signal regions and relative normalizations of backgrounds.

Production of \(W/Z+\)jets with subsequent leptonic decays of \(W\) and to a much smaller degree \(Z\) is also a substantial source of background for SR1 and SR2 when the resulting charged leptons fail to be identified or if the \(W\) or \(Z\) bosons decay to \(\tau \) leptons. These contributions are estimated from \(Z(\ell ^+ \ell ^-)+\)jets and \(W(\ell \nu )+\)jets MC samples generated using ALPGEN2.3 [46] with the CTEQ6L1 [47] parton distribution function (PDF) set. The procedure used for the normalization of this sample is described in reference [48]. These samples are generated with up to five light partons (\(u,~d,~s\)) and one \(c\) quark or two heavy quarks (\(c\), \(b\)) per event. \(W+b\) production is highly suppressed and therefore negligible. A control region enriched in \(W(\ell \nu )+\)jets events is selected by adding a lepton requirement to the selection and is used to validate the estimate of this background. The purity of \(W(\ell \nu )+\)jets in the control region for SR1 (SR2) is 67 % (47 %). After full selection the contribution of \(b\)(\(c\))-quarks to the dominant \(W(\ell \nu )\)+jets background is approximately 39 % (38 %) for SR1 and 52 % (37 %) for SR2. The systematic uncertainty on this background is approximately 20 %. Finally, the small contribution from \(t\bar{t}\) is estimated using MC samples and validated in data control regions before applying signal selection requirements. The \(t\bar{t}\) process is selected with very high purity by requiring events with one lepton and large jet multiplicities.

Fig. 4
figure 4

Lower limits on \(M_*\) at 90 % CL for the SR1 (red), SR2 (black), SR3 (green), and SR4 (blue) as a function of \(m_{\chi }\) for the operators a D1, b C1, and c D9. Solid lines and markers indicate the validity range of the effective field theory assuming couplings \(g_q g_\chi <4\pi \), the dashed lines and hollow makers represent the full collider constraints

The dominant source of background for SR3 and SR4 is \(t\bar{t}\) events. In SR3, this contribution is estimated from data using a control region not overlapping with SR4 and largely dominated by \(t\bar{t}\) events with one of the two top quarks decaying semileptonically. The five-jets requirement is relaxed to three jets. Additionally, the event is required to contain exactly one lepton with \(p_\mathrm{T}^{e(\mu )}>30~(25)\) GeV and must fulfill \(E_\mathrm {T}^{\mathrm {miss}}+m_\mathrm {T}>25~(30)\) GeV for the electron (muon) channel. The potential signal contribution to this selection is less than 0.1 %. The uncertainties are small because the SR3 data control region uses a kinematic region similar to the signal region with the lepton veto and jet multiplicity being the main difference. These effects were studied and considered as systematic uncertainties. Dominant uncertainties are related to jets and the top quark momentum distribution. Corrections to compensate for the differences in efficiency and acceptance between hadronic and semi-leptonic top decays are derived from MC samples generated using the POWHEG BOX generator [49] interfaced with JIMMY4.31 [50] with the next-to-leading-order (NLO) PDF set CT10 [51]. The systematic uncertainty on the \(t\bar{t}\) background in SR3 of approximately \(7\,\%\) is derived by studying corrections for the top quark momentum distribution, and shower modelling by interfacing the same generator with PYTHIA6 [52, 53].

In SR4, the \(t\bar{t}\) background is estimated from data using a control region obtained by requiring \(60~\text {GeV}< m_\text {T}< 90\) GeV and loosening the selection criteria on \(E_\mathrm {T}^{\mathrm {miss}}\), \(am_\mathrm {T2}\), and \(E_\mathrm {T}^{\mathrm {miss}}/ \sqrt{H_\text {T}^{4j}}\). A similar selection, but applying an inverted \(b\)-tagging requirement, is used to estimate the \(W(\ell \nu )+\)jets background. The uncertainty on the \(t\bar{t}\) background is estimated to be approximately 20 % [24], which is larger than the uncertainty in SR3 due to the limited statistics. These uncertainties are evaluated by varying the renormalisation and factorisation scale of the simulations, comparing alternative PDF sets, and studying the effects of different shower generators and of ISR and final-state radiation.

Additional sources of background, which include single-top, \(t\bar{t}+Z/W\), and diboson production, are estimated in all signal regions using simulations and NLO cross sections [54, 55]. The single-top (s-channel) and \(Wt\) background is generated using the POWHEG generator. The single-top t-channel is generated with ACERMC3.8 [56] interfaced with PYTHIA6. Associated production of \(t\bar{t}\) and a vector boson (\(W\), \(Z\)) are generated with MADGRAPH5 [57] with up to two additional partons interfaced with PYTHIA6. The cross-sections for \(t\bar{t}\) production in association with a \(W\) (\(Z\)) boson are determined using the MSTW2008 NLO (CTEQ6.6M) PDF sets. The diboson samples are generated using HERWIG6.520 [58, 59] and JIMMY4.31 with the CTEQ6L1 PDF set. The multijet background is estimated using data-driven methods [60] and is found to be negligible in all signal regions after full selection.

Object reconstruction efficiencies in simulated events are corrected to reproduce the performance measured in data. The systematic uncertainty of the background estimates derived from simulation combines the uncertainties on the efficiency of the \(b\)-tagging algorithm, the uncertainties on the determination of the energy scale and resolution of the jet energy and \(E_\mathrm {T}^{\mathrm {miss}}\), the theoretical uncertainty on the various cross-sections, changes in the shapes of distributions used to extrapolate event counts from control regions to the signal region, data driven corrections and the PDF uncertainties. Overall, the systematic uncertainty on the background estimated from simulation is calculated to be between 12 and 18 %, depending on the signal region.

The simulation of the signal samples of \(pp\rightarrow \chi \overline{\chi }+ b(\overline{b})\), \(pp\rightarrow \chi \overline{\chi }+ t\overline{t}\), and \(b\)-FDM employs the MADGRAPH5 generator interfaced with PYTHIA6 using the CTEQ6L1 PDF. Samples are generated for operators D1, C1, and D9, assuming \(M_*= 1\)  TeV and \(m_{\chi }\) between 10 and 1300 GeV. Samples for the \(b\)-FDM model are generated for \(m_\chi \) values between 1 and 1300 GeV and mediator masses, \(m_{\phi }\), between 5 and 3000 GeV. The instrumental uncertainties on the simulated signal yields for D1, C1, and D9 operators are between 11 and 15 %, depending on the signal region. The equivalent uncertainties for the \(b\)-FDM model range between 6 and 16 % depending on \(m_{\chi }\) and the mediator mass. The uncertainties from the PDF are computed by comparing the rates obtained with the default PDF set (CTEQ6L1) with those obtained with two alternative sets (MSTW2008LO and NNPDF21LO [61, 62]). The uncertainties on the signal acceptance from PDF and scale variations are estimated to be approximately 10 % for the D1, C1, and D9 operators for \(m_{\chi }=10\) GeV and approximately \(6\) % for \(b\)-FDM models.

The validity of the effective field theory assumption depends on the momentum transfer of the process modelled, which should be below the energy scale of the underlying interactions [63]. To account for this, the momentum transfer \(m(\chi \chi ) = Q_\mathrm {tr}\) in the events is required to be less than the energy scale probed. Specifically, \(Q_\mathrm {tr}\) must be smaller than the mass \(M\) of the heavy mediator. For an ultraviolet completion this implies \(M_*=M/\sqrt{g_q g_\chi }\). Along with perturbativity of the couplings \(g_q g_\chi <4\pi \) this leads to the following validity requirements on MC truth level: \(Q_\mathrm {tr}<4\pi ( M_*^3/m_q )^{1/2}\) (D1), \(Q_\mathrm {tr}<4\pi M_*\) (D9), \(Q_{\mathrm tr}<(4 \pi )^2 M_*^2 / m_q\) (C1).

4 Results

Table 2 shows the expected background from various sources in the four signal regions as well as the observed yields in data. The expected signal yields for the operators D1, C1, and D9, as well as for the \(b\)-FDM model are also shown. The probabilities of the background-only hypothesis, \(p\) values, for the signal regions SR1, SR2, SR3, and SR4 are 0.09, 0.29, 0.24, and 0.18, respectively. As no significant excess is observed, limits on the signal yield are set using a profile likelihood ratio test following the \(CL_s\) prescription [64]. Also given is the 95 % confidence level (CL) upper limit on the number of beyond-the-SM events. The yields for the \(b\)-FDM model are obtained assuming \(m_\chi =10\) GeV and a mediator mass \(m_\phi =600\) GeV. The limit on \(M_*\) for a given assumption on \(m_\chi \) is determined by varying \(M_*\) and scaling the number of signal events predicted by the corresponding sample generated with \(M_* = 1\) TeV until it is equal to the observed upper limit on beyond-the-SM events. The corresponding production cross-section for DM produced via the D1 operator in association with \(b\)(\(t\))-quarks and \(m_\chi =10\) GeV is 38 (221) fb. The cross-section for \(b\)-FDM models with \(m_\phi =600~\) and \(m_\chi =10\) GeV is 134 fb. The signal efficiency is independent of \(M_{*}\).

Figure 3 shows the \(E_\mathrm {T}^{\mathrm {miss}}\) distributions for (a) SR1, (b) SR2, and (d) SR4 and (c) the \(R\) variable for SR3.

Fig. 5
figure 5

Upper limits at 90 % CL on the spin-independent \(\chi \)–nucleon cross-section (\(\sigma _{\chi -\hbox {N}}^\mathrm{SI}\)) for the scalar operator D1 (red) as a function of \(m_{\chi }\). The yellow and green curves represent the exclusion limits recently set by the LUX and Super-CDMS collaborations [6, 7, 65]. The coupling is assumed to be \(g_q g_\chi =g=4 \pi \)

Fig. 6
figure 6

Upper limits at 90 % CL on the spin-dependent \(\chi \)–nucleon cross-section (\(\sigma _{\chi -\hbox {N}}^\mathrm{SD}\)) for the tensor operator D9 (red) as a function of \(m_{\chi }\). The yellow and green curves represent the exclusion limits recently set by the COUPP and PICASSO collaborations [8, 9, 65]. The coupling is assumed to be \(g_q g_\chi =g=4 \pi \)

Fig. 7
figure 7

Exclusion contour at 95 % CL for the \(b\)-FDM model from combined results of SR1 and SR2. The expected limit is given by the dashed line, and the yellow band indicates the \(\pm 1~\sigma \) uncertainty. The observed limit, largely dominated by SR1, is given by the solid red line. The region beneath the curve indicating the observed limit is excluded

Figure 4 shows the 90 % CL exclusion curves for the effective mass scale \(M_*\) as a function of \(m_{\chi }\). The results for the operators D1, C1, and D9 are presented individually for all four signal regions. The best limits on the D1 and C1 operators are obtained using SR4, while SR1 provides the best limits on the D9 operator, as shown in Fig. 4. These limits are then converted into limits on the \(\chi \)–nucleon cross-section [12]. Figures 5 and 6 show the corresponding 90 % CL exclusion curves for the spin-independent and spin-dependent \(\chi \)–nucleon cross-section for the scalar (D1) and tensor (D9) operators as a function of \(m_{\chi }\) for the strongest results obtained in any signal region. The most stringent limits set by direct detection experiments [69] are also shown. Only \(m_\chi \) where more then \(90\,\%\) of the events fulfill the effective field theory validity constraints are shown in Figs. 5 and 6.

The limits shown are especially strong in the low-mass region where several collaborations [28, 6668] have recently claimed possible observations of DM. The results reported in this article represent the first ATLAS limits on the scalar operator C1 and they significantly improve the sensitivity to \(\chi \)–nucleon interactions mediated by the scalar operator D1 compared to previous ATLAS results [14, 16, 18, 19].

Figure 7 shows the exclusion curves observed and expected for the \(b\)-FDM model as a function of the mediator and DM masses. For each point in (\(m_\chi \), \(m_\phi \)), the signal region with the best expected sensitivity is used, with SR1 dominating over the other signal regions. For a DM particle of approximately 35 GeV, as suggested by the interpretation of data recorded by the Fermi-LAT collaboration, mediator masses between approximately 300 and 500 GeV are excluded at 95 % CL.

5 Conclusions

In summary, this article reports a search for dark-matter pair production in association with bottom or top quarks. The analysis is performed using \(20.3 \mathrm {~fb}^{-1}\) of \(pp\) collisions collected at \(\sqrt{s} = 8\) TeV by the ATLAS detector at the LHC. The results are interpreted in the framework of an effective field theory to set stringent limits on scalar and tensor interactions between Standard Model and DM particles. The data are found to be consistent with the Standard Model expectations, and limits are set on the mass scale of effective field theories that describe scalar and tensor interactions between DM and Standard Model particles. The exclusion limits are strongest at low DM masses. The limit on the \(\chi \)–nucleon cross-section mediated by the D1 operator is improved significantly with respect to previously published ATLAS results by obtaining sensitivities of approximately \(\sigma _{\chi -\hbox {N}}^\mathrm{SI}=10^{-42}\,\text {~cm}^2\) for \(m_\chi =10\) GeV. Constraints on \(b\)-Flavoured Dark Matter models, suitable to explain a possible signal of annihilating DM, are also presented. The excluded regions depend on \(m_\chi \) and \(m_\phi \). For \(m_\chi =35\) GeV, mediator particles with \(m_\phi =300\)\(500\) GeV are excluded.