1 Introduction

Neutrinos, which correspond to a major portion of the elementary particles comprising the Universe, are still poorly understood. In the standard model (SM) of particle physics, neutrinos are assumed to be massless particles with 1 / 2 spin. However, the recent discovery of neutrino oscillations [1,2,3,4] demonstrates that the flavour states of neutrinos are linear combinations of mass eigenstates characterized by the Pontecorvo–Maki–Nakagawa–Sakata (PMNS) matrix [5, 6]. These results clearly demonstrate that neutrinos have non-zero mass; the mass eigenstates’ different time evolutions produce the observed oscillations between different flavour eigenstates.

The differences of the squares of the mass eigenvalues are experimentally determined from the oscillation frequencies of solar, atmospheric, accelerator and reactor neutrinos [1,2,3, 7]. However, the absolute mass scale still remains unknown. Moreover, it has still not been determined if neutrinos are Majorana particles, in which case neutrinos are their own antiparticles, or Dirac particles, in which case the neutrinos and antineurinos are distinct. This is a fundamental feature of the SM that remains unanswered [8].

In two-neutrino double beta decay (\(2\nu \beta \beta \)), expressed as

$$\begin{aligned} (Z,A)\rightarrow (Z+2,A)+2 \beta ^- + 2 \bar{\nu }_e, \end{aligned}$$

there is simultaneous emission of two electrons and two anti-neutrinos. These processes have been observed for a number of nuclides [9, 10]. On the other hand, neutrinoless double beta decay (\(0\nu \beta \beta \)), which is expressed as

$$\begin{aligned} (Z,A)\rightarrow (Z+2,A)+2 \beta ^- ~, \end{aligned}$$

is a hypothetical process that is allowed if neutrinos are Majorana particles with a non-zero mass. Since no anti-neutrinos are emitted in this process, it violates lepton number conservation by two units [8, 11,12,13] and the total available decay energy is shared by the two electrons, with a negligible amount transferred to the daughter nucleus. Consequently, \(0\nu \beta \beta \) decay would produce a distinct narrow peak at the end-point of the broad electron energy spectrum of the \(2\nu \beta \beta \) decay.

The SM is invariant under a U(1) gauge transformation, leading to the lepton number conservation [14]. Searching for \(0\nu \beta \beta \) decay is currently the only plausible experimental technique to test the Majorana nature of neutrinos and the validity of lepton number conservation. The discovery of this decay would provide major new insights into the nature of beyond-the-SM physics.

In general, the \(0\nu \beta \beta \) decay rate can be expressed as

$$\begin{aligned} {[}T^{0\nu }_{1/2}]^{-1} = G_{0\nu } \left| M_{0\nu } \right| ^2 \left| \eta \right| ^2, \end{aligned}$$

where \(T^{0\nu }_{1/2}\) is the half-life of the \(0\nu \beta \beta \) decay, \(G_{0\nu }\) is the phase space factor for the decay [15, 16], \(M_{0\nu }\) is the nuclear matrix element [17], and \(\eta \) is a parameter that depends on the mechanism of the process that mediates the lepton number violation process. If the process is mediated by light neutrino exchange, \(\eta = \langle m_{\beta \beta }\rangle \), which is the effective Majorana mass expressed as the coherent sum

$$\begin{aligned} \langle m_{\beta \beta }\rangle = \left| \sum ^3_{i=1} U^2_{ei} m_i \right| , \end{aligned}$$

where \(U_{ei}\) are PMNS matrix elements, and \(m_i\) are the neutrino mass eigenstates. Note that Eq. (3) is the standard interpretation of the general lepton number violation processes. Other mechanisms that lead to \(0\nu \beta \beta \) at the same order strength as light neutrino exchange have been proposed [18]. Therefore, a full understanding of the \(0\nu \beta \beta \) process would require \(0\nu \beta \beta \) measurements for more than one nuclide to distinguish between the different possibilities. In any case, a \(0\nu \beta \beta \) discovery would be a clear demonstration of the Majorana nature of neutrinos and of the lepton number violation. Moreover, the nuclear matrix values determined using different theoretical calculations differ by factors as large as two [17]. These uncertainties also call for \(0\nu \beta \beta \) search experiments that use different nuclides [19].

Table 1 A summary of the most sensitive \(0\nu \beta \beta \) decay search experiments. The target isotopes and the current half-life limits are shown
Fig. 1
figure 1

Figure from  [35]

A detector module for AMoRE-Pilot experiment. It is comprised of three main parts: a doubly enriched \(^{48{{\text {depl}}}}\hbox {Ca}^{100}\hbox {MoO}_{4}\) crystal, a photon detector, and phonon detector.

To date, \(0\nu \beta \beta \) decay has not been observed; the most sensitive experiments give lower limits on decay half lives at the level of \(10^{24}\) to \(10^{26}\) years. The current best limit on any \(0\nu \beta \beta \) decay rate was established for \(^{136}\)Xe by the KamLAND-Zen collaboration, which used xenon-loaded liquid scintillator in a balloon and set a half-life limit of \(0\nu \beta \beta \) at \(1.07\times ~10^{26}\) years [20]. Germanium detector arrays made of \(^{76}\)Ge are used by GERDA [21] and the Majorana-Demonstrator [22, 23]. A liquid xenon time projection chamber (TPC) is employed by EXO-200 [24]. CUORE searches for \(^{130}\)Te \(0\nu \beta \beta \) decay using thermal calorimeters operating at mK temperatures [25]. Recently, CUPID-0 carried out a \(^{82}\)Se \(0\nu \beta \beta \) decay search using the simultaneous heat and scintillation detection technique [26]. NEMO used electron trackers and plastic scintillators for \(0\nu \beta \beta \) decay searches with various isotopes. The most recent version of the experiment, NEMO-3, obtained the most stringent half-life limit on \(^{100}\)Mo \(0\nu \beta \beta \) decay of \(1.1\times 10^{24}\) years [27]. Some of the most sensitive \(0\nu \beta \beta \) decay search experiments are summarized in Table 1.

The advanced molybdenum-based rare process experiment (AMoRE) [28, 29] aims to search for \(0\nu \beta \beta \) decay of \(^{100}\)Mo using a simultaneous heat and scintillation detection technique with crystals operating at mK temperatures. In the present stage of the project, AMoRE-Pilot, calcium molybdate (\(\hbox {CaMoO}_{4}\)) crystals are used as the active target elements in a detector concept where the source material is also the detection medium [30,31,32]. The crystals are \(^{100}\)Mo-enriched and \(^{48}\)Ca-depleted (\(^{48{{\text {depl}}}}\hbox {Ca}^{100}\hbox {MoO}_{4}\)). Since \(^{48}\)Ca is a double beta decaying isotope with a Q value of 4271 keV, its \(2\nu \beta \beta \) decay energy spectrum overlaps the 3034 keV energy region of interest (ROI) for AMoRE’s \(0\nu \beta \beta \) decay search, producing an irreducible background at the level of \(\sim \) 0.01 counts/(keV kg year) [30]. Thus, the amount of \(^{48}\)Ca must be reduced.

Six \(^{48{{\text {depl}}}}\hbox {Ca}^{100}\hbox {MoO}_{4}\) crystals with a total mass of 1.9 kg are currently in operation at the 700-meter-deep Yangyang underground laboratory (Y2L) located in South Korea. A cryogen-free dilution refrigerator (CF-DR) is used to cool down the AMoRE-Pilot detector array to 10–20 mK, including sensors for heat and scintillation detection. The data sample used for the results reported here corresponds to a 52.1 kg day live exposure of \(^{100}\hbox {Mo}\).

2 Experimental setup and data acquisition

The AMoRE-Pilot detector modules are comprised of three main components, as shown in Fig. 1: a \(^{48{{\text {depl}}}}\hbox {Ca}^{100}\hbox {MoO}_{4}\) scintillating crystal, a phonon sensor based on a metallic magnetic calorimeter (MMC) that measures the temperature rise of the crystal induced by radiation absorption, and a photon detector with an MMC sensor that detects the amount of scintillation light produced in the crystal [33,34,35].

The crystals are assembled inside a high radiopurity, high conductivity copper frame that also functions as a thermal bath. Three conically shaped polyether-ether-ketone (PEEK) spacers placed beneath the crystals provide support with minimal thermal losses. Four copper tabs on the top of each frame firmly press down the crystals and keep them stationary. Polytetrafluoroethylene (PTFE) spacers are installed between the holders and the crystal to reduce heat losses. The crystals are elliptical cylinders of different sizes, with masses that range from 196 to 390 g.

The phonon sensor is situated at the bottom of the detector-module copper frame. A 400 nm thick gold film deposited on the lower surface of the crystal serves as a phonon collector. This is connected to an MMC by 25 annealed gold wires [36]. The MMC exploits the paramagnetic nature of erbium ions in a gold host (Au:Er) [37]. The devices were micro-fabricated on a Si wafer by sputtering a planar niobium coil with a layer of Au:Er sensor material immediately above the coil [38, 39]. A persistent current in the superconducting coil generates a magnetic field in the Au:Er sensor material, and energy deposited in the crystal produces a temperature increase in the Au:Er that alters its magnetization. This results in a change in the coil current that is measured by a superconducting quantum interference device (SQUID) [31]. Because of the high sensitivity of the MMC sensors, the detector setup is able to measure precisely the temperature increase resulting from any type of radiation absorption, and particle detection with very high energy resolution is achievable [35, 40].

A detachable photon detector [41] is installed at the top of the copper frame. The photon detector is comprised of a 2-inch diameter, 300 \(\upmu \)m thick germanium wafer that serves as a scintillation light absorber; an MMC sensor measures the phonon energy generated inside the germanium wafer. With this arrangement, we measure the phonon signals and scintillation light signals simultaneously; by comparing the relative amplitudes of the simultaneous signals, we can discriminate between the unwanted non-\(\beta /\gamma \)-generated events and the \(\beta /\gamma \)-like events of interest [33,34,35].

The AMoRE-Pilot experiment involved five data-taking runs between 2015 and 2017 [32]. The goals of the first four runs were to: realize the measurement setup with a refrigerator system and a lead shield system in the underground laboratory; investigate the noise and the background levels; and improve the detector performance. During these research and development stages, we used a CF-DR with a 7 mK minimum temperature in a newly built laboratory room in Y2L. As shown in Fig. 2, we also assembled a detector array comprised of several detector modules below the 10 mK plate. For the first five runs, five \(^{48{{\text {depl}}}}\hbox {Ca}^{100}\hbox {MoO}_{4}\) crystal detector modules with a total mass of 1.547 kg (0.727 kg of \(^{100}\hbox {Mo}\)) were operated. During this R&D phase, we significantly improved the detectors performance. We installed a 15 cm-thick lead shield to cover the top and the sides of the CF-DR to suppress background radiation from external sources. An active muon veto system (MVS) was used to tag and reject muon-induced background events. Vibration noise from the pulse tube refrigerator of the CF-DR, initially the dominant source of signal noise, was reduced by a few orders of magnitude by means of a two-stage vibration isolation system consisting of a spring suspended still (SSS) with eddy current dampers (ECDs) [42] and a mass-spring damper that mechanically isolates the detector tower from the 10 mK CF-DR mixing chamber plate (see Fig. 2). During the fifth run, the SSS mechanism was not fully operational because of damage to one of its magnets. However, it still had outstanding vibration reduction performance that improved the noise conditions by several orders of magnitude [42]. The SSS system was in full operation for the subsequent runs.

Fig. 2
figure 2

Schematic diagram of the AMoRE-Pilot detector system

The fifth run of AMoRE-Pilot, the main data collection run of the experiment, started operation in August 2017. For this, we added a sixth \(^{48{{\text {depl}}}}\hbox {Ca}^{100}\hbox {MoO}_{4}\) crystal of mass 340 g, thereby increasing the total crystal mass to 1.887 kg (0.886 kg of \(^{100}\hbox {Mo}\)). The six detector modules were arranged in a single tower inside a superconducting lead shield. The detector tower and shield were maintained at a temperature of 20 mK throughout the data collecting period.

Data were acquired by a 12 channel 18-bit ADC, including six phonon and six photon channels. A continuous stream of data was saved and later triggered off-line via the application of a Butterworth filter [43, 44]. The full dynamic range of the measurement was ± 3 V, and limited by a differential amplifier. The sampling rate of the DAQ was set to 100 kHz, which is sufficient for pulse shape discrimination (PSD) of different types of background events based on the rise time of the phonon signals. During this phase of the experiment, we collected a 69.0 kg day \(^{100}\hbox {Mo}\) exposure. The six detector channels and the cryogenic system operated reliably throughout the approximately 1000 h measurement period.

3 Primary data processing

A Butterworth bandpass filter was applied to both of the original phonon and photon signals to enhance the signal-dominant frequency band and suppress noise-dominant frequency bands [44]. We evaluated many sets of passbands and chose the set that provided the highest signal-to-noise ratio (SNR) for each channel. The filtered signals were also well suited for effective triggers [44]. The energy threshold was between 2 and 5 keV depending on the crystal; the off-line trigger levels were set to several tens of keV for the \(0\nu \beta \beta \) decay search analysis; the Q value of \(^{100}\)Mo \(0\nu \beta \beta \) decay is \(\sim 3~\hbox {MeV}\).

To determine the phonon and photon signal amplitudes for each event, the filtered signals were fitted by a least squares method that minimized the sum of squared residuals with a scaled filtered template developed with 2615 keV gamma ray signals from \(^{208}\)Tl contaminants in the crystals. This is the highest naturally occurring intensive \(\gamma \) peak and the closest one to the ROI. For a signal vector \(S_i\) in the time domain and the template signal vector \(T_i\) of length N, scaled with the scale factor A, the squared error to the template (SE) is written as:

$$\begin{aligned} SE = \Sigma _{i = 1}^N (S_i - A T_i)^2. \end{aligned}$$

The scale factor A that minimizes SE is the amplitude of the signal normalized to the template:

$$\begin{aligned} A = \frac{\Sigma _{i = 1}^NS_i \cdot T_i}{\Sigma _{i = 1}^NT_i^2}. \end{aligned}$$

Here we assume that all the \(\beta /\gamma \)-signals have the same shape as the template, but with different amplitudes. Divergence from this assumption produces a small non-linearity in the amplitude-to-energy calibration. The amplitudes of non-\(\beta /\gamma \)-like events are also calculated with this method, but their energy scale is different because of the mismatch between their pulse shape and that of the template.

Signal amplitudes can be influenced by a long term instability of the temperature. In the present system, it appears as a long-term thermal gain drift that can be corrected with frequent reference to events with a known energy. For AMoRE-Pilot, one of the six crystals, Crystal 2 (see Table 2), has a high internal \(^{210}\)Po \(\alpha \) rate. These events were used as standard signals to provide gain stabilization signals for all six crystals. The application of the gain stabilization for one detector using events from another one was demonstrated in previous AMoRE-Pilot runs. For future runs, gain corrections will be improved with stabilization heaters installed on each crystal [45].

The energy calibration of each crystal was done with a \(^{232}\hbox {Th}\,\gamma \) source rod placed in between the lead shield and the outer vacuum chamber of the cryostat for one day each week. The background spectrum was compared to the calibration spectrum for peak identification. We then made use of between 7 and 11 strong \(\gamma \)-peaks to determine the signal amplitude vs the energy calibration.

Our earlier work suggested that the amplitudes of \(\beta /\gamma \) peaks follow a simple quadratic function of energy up to 2615 keV [34, 35]; we extrapolate this to higher energies. The suitability of a quadratic polynomial with no constant term for energies above 2615 keV was verified by fitting \(\alpha \)-peaks, including the \(2311~\hbox {keV}\,^{147}\hbox {Sm}\) peak, with another quadratic polynomial function. Although \(\alpha \)- and \(\beta /\gamma \)-events generate pulses of different amplitudes due to the different signal shapes, good agreement between \(\alpha \)- calibration lines and a quadratic polynomial provides evidence that a similar detector response is adequate for \(\beta /\gamma \)-events with energies above 2615 keV.

Table 2 DP values and the effective selection efficiency (See Sect. 5.1) by using both the L/H and RT cuts at the Q value of \(^{100}\)Mo decay (3034 keV)
Fig. 3
figure 3

Top: energy spectrum of one \(^{48{{\text {depl}}}}\hbox {Ca}^{100}\hbox {MoO}_{4}\) detector for a 60 day exposure after the application of the event selection criteria, without a calibration source. The most prominent \(\gamma \) peaks are labelled. Bottom: a second-order polynomial calibration curve (solid) with its 95% P.I. (dashed). The vertical dashed line in the bottom figure indicates the Q value of \(^{100}\)Mo double beta decay. See Sect. 4 for a detailed explanation of the event selection methods

Fig. 4
figure 4

\(\gamma \) peaks from a background spectrum of one of the \(^{48{{\text {depl}}}}\hbox {Ca}^{100}\hbox {MoO}_{4}\) detectors. Top: 609 keV from \(^{214}\hbox {Bi}\). Middle: 1332 keV from \(^{60}\hbox {Co}\). Bottom: 2615 keV from \(^{208}\hbox {Tl}\). Solid lines show the best fit of the gamma peaks by exponentially modified Gaussian (see text) with an exponential function to describe the continuous background. The skewness increases with the energy

Figure 3 shows a background energy spectrum from one of the crystals that shows signal amplitudes that cover the energy range below 2615 keV, and displays a non-linear behaviour and deviations from the calibration. The solid line shows the calibration. The negative slope reflects the non-linear term. The peak positions are in good agreement with the 95% calibration prediction interval (P.I.) shown as dashed lines, indicating that the energy of a signal can be estimated using the calibration and error over a wide energy region. Uncertainties in the energy calibration are included in the likelihood marginalization [46].

Figure 4 shows shapes of \(\gamma \) peaks with energies 609, 1332 and 2615 keV in the data accumulated with one of the \(^{48{{\text {depl}}}}\hbox {Ca}^{100}\hbox {MoO}_{4}\) detectors. The 1332 keV peak originated from \(^{60}\hbox {Co}\) in the outer vacuum chamber. The solid curves are the fits of the energy (E)-dependent model function of the detector response to mono-energetic \(\gamma \)-rays in the presence of exponential backgrounds:

$$\begin{aligned} \begin{aligned}&f(E;A,\mu ,\sigma ,\lambda ,a,b) \\&\quad = \xi \cdot f_{{\text {EMG}}}(E;\mu ,\sigma ,-\lambda ) + a \cdot {{\text {exp}}}(-b \cdot E). \end{aligned} \end{aligned}$$

Here \(f_{{\text {EMG}}}(E;\mu ,\sigma ,-\lambda )\) is an exponentially modified Gaussian with a mean \(\mu \), standard deviation \(\sigma \) and negative exponential component \(-\lambda \):

$$\begin{aligned}&f_{{\text {EMG}}}(E;\mu ,\sigma ,-\lambda )\nonumber \\&\quad = -\frac{\lambda }{\sqrt{\pi }} {{\text {exp}}}\left( -\frac{\lambda }{2}\left( 2 \mu - \lambda \sigma ^2 - 2 E\right) \right) \int ^\infty _E {{\text {exp}}}(-t^2) {{\text {d}}}t,\nonumber \\ \end{aligned}$$

where \(\xi \) is the signal strength parameter, and a and b are parameters from an exponential representation of the continuous background. The EMG function is used to model the low energy tails of \(\gamma \) and \(\alpha \) peaks caused by incomplete phonon collection in the detectors. The skewness \(1/ \lambda \) was set to be proportional to the mean (\(\mu \)) of the model function. Fits using this prescription to many peaks with a single value for \(E/\lambda \) give good fits to the measured peaks for all the crystals and is in the model function for \(0\nu \beta \beta \) events.

4 Event selection

We used a variety of event selection methods to improve the signal to background ratio in the ROI. The selection criteria are based on operational status, event-by-event identification, and coincidence tagging.

The operational status selection excludes the data-taking periods where measurements were inactive or unstable. The rejected periods include times for system maintenance, calibration, and periods of excessive noise and sensor instabilities; and dominate the total dead-time, resulting in a live-time that is 84% of the total data-taking time interval.

Fig. 5
figure 5

Scatter plot of the light yield vs energy deposited in the phonon channel for one crystal. The \(\beta /\gamma \) band and the \(\alpha \) clusters are clearly separated, as labelled in the figure

The event-by-event identification selection criteria discriminate \(\beta /\gamma \)-induced events from other background signals: \(\alpha \)-induced signals; SQUID resets, which appear as a sudden change in DC value by integer multiples of a unit magnetic flux quantum; anomalous signals that have poor template fits as indicated by large minimum SE values; and high energy events that exceed the dynamic range. These were implemented by using two particle identification (PID) techniques. One is from the relative amplitude ratio between light and heat signals (L/H ratio) in the simultaneous phonon-scintillation measurement and the other is the pulse shape discrimination (PSD) of the heat signals.

The underlying concept of the L/H ratio method is that the amount of scintillation light emitted by a scintillator as a result of the interaction with a charged particle depends inversely on its stopping power [47]. An event induced by a \(\beta /\gamma \) deposit in a crystal yields a larger light signal than one induced by an \(\alpha \) with the same energy, resulting in different L/H ratios. Figure 5 displays a 2D scatter plot for all events after the operational status and non-physical signals rejections, where clear \(\beta /\gamma \) and \(\alpha \) separation is evident.

The pulse shape of the phonon signal can also be used for event identification. The \(\hbox {CaMoO}_{4}\) bolometric signal rise time (RT), the time difference between 10%–90% of the pulse maximum, is different for \(\beta /\gamma \)- and \(\alpha \)-induced events [33, 35, 48, 49]. In both the L/H ratio and RT distributions, some events appear in between the two distinct groups. These events are mostly caused by random signal overlaps (pileups) and/or poorly reconstructed signal shapes.

Using either L/H or RT values for event identification, a discrimination power (DP) for background rejection can be defined as:

$$\begin{aligned} {{\text {DP}}} = \frac{|\mu _{\beta /\gamma }-\mu _{\alpha }|}{\sqrt{\sigma _{\beta /\gamma }^2 +\sigma _{\alpha }^2}}, \end{aligned}$$

where the \(\mu \)’s and \(\sigma \)’s are the mean values and the standard deviations of those events distributions, respectively. For the distributions of the parameters \(\mu \) and \(\sigma \), the two techniques have different energy dependence. Assuming the light yield per keV energy deposit in a crystal is independent of the deposited energy, we can infer a constant value for the \(\mu _{\beta /\gamma }\) parameter of the L/H ratio. The validity of this assumption was established in previous reports [34, 35]. The other values\(-\sigma \) for the L/H discriminator, and \(\mu \) and \(\sigma \) for the RT discriminator −depend on the energy. The DP values of the two event identification parameters, determined for energies at the Q value of \(^{100}\)Mo decay, are listed in Table 2.

For \(\beta /\gamma \) events, we assume a constant mean and dispersion of the L/H ratio. The measured distribution was divided into narrow, 100 keV energy slices between 465 and 2665 keV, and Gaussian fits were used to determine 24 different values of \(\sigma _{\beta /\gamma }\). The energy dependent upper and lower bounds, \(g_{\pm }\), on the L/H ratio are parametrized by

$$\begin{aligned} g_{\pm }(E) = \pm c_1~{\text {exp}}( - c_2~E) + c_3, \end{aligned}$$

with \(c_i\) values determined from fits to the L/H ratio distribution. The function determined from the measured values of \(\mu \pm 2 \sigma _{\beta /\gamma }\) is shown as a green band in Fig. 5. The events inside the band are considered as \(\beta /\gamma \) events.

Fig. 6
figure 6

Scatter plot of the rise time vs energy deposited in a phonon channel. The \(\beta /\gamma \) band and the \(\alpha \) clusters are clearly separated, as labelled in the figure

A similar method was applied to the RT parameter, as shown in Fig. 6. Since the pulse shape for the phonon channel has some energy dependence, a linear term was added to the mean and the dispersion of the RT distribution. Using \(\mu \pm 2\sigma _{\beta /\gamma }\) of the RT distribution as upper and lower limits, the characteristic function \(h_{\pm }\) of the dispersion is:

$$\begin{aligned} h_{\pm }(E) = \pm d_1~{\text {exp}}( - d_2~E) + d_3 E + d_4, \end{aligned}$$

where \(d_i\) are fitting parameters.

The third class of event selection requirements, coincidence tags, removes multiple-hit events and pairs of decays correlated in time. While many background events can deposit energy across multiple locations, double beta decay events are contained within short ranges of the emitted electrons, confining the energy deposition to a single crystal. Thus a multiple-hit selection was applied to reject events with signals in more than one crystal within a 2 ms coincidence window. The MVS surrounding the external lead shield provided the capability for rejecting signals induced by cosmogenic muons. Approximately 2000 muons/day are detected in the MVS. A veto period of 100 ms was applied following each muon signal to reject any coincident signals in the crystals. During the pilot stage, the MVS did not provide the full 4-\(\pi \) coverage over the detector modules. The muon rejection efficiency was determined to be between 87 and 92%, depending on the crystal location. Finally, we reject background events from internal \(^{208}\)Tl \(\beta \) decays, which have a Q value of 4999 keV, by tagging the prior 6207 keV emission from the associated \(\alpha \) decay of \(^{212}\)Bi. The time between the two decays is characterized by 3.05-min half-life of \(^{208}\)Tl \(\beta \) decay and, thus, a veto-time of 30 min was set following the \(^{212}\)Bi \(\alpha \) detection. This dead time only applies to a single crystal. The good energy resolution and identification capabilities of the crystals allowed the \(^{212}\)Bi events to be tagged with high specificity. With this method, the \(^{208}\)Tl rejection efficiency in the ROI was 97.5%. In the present analysis, the dead time from the 30 min \(^{212}\)Bi tagging was 1.7%, as indicated in Table 3. Since the phonon signal duration is approximately 1 s, we applied a trigger dead time of 1 s for each of the triggered events. The exposures that survived each event selection requirement are summarized in Table 3.

The energy spectrum in the 2850–3150 keV ROI for the 111 kg day exposure that remains after the application of the event selection requirements and coincidence tagging vetoes is shown in Fig. 7. The spectrum shape of the 50 events in the plot is approximated by a linear background function plus the EMG function for the \(0\nu \beta \beta \) peak. The width of this energy region was chosen to cover the total response function for \(0\nu \beta \beta \) events while avoiding the influence of the 2615 keV \(^{208}\)Tl peak.

The systematic errors associated with the exposure include the uncertainties of mass measurement and \(^{100}\hbox {Mo}\) content estimations. The crystals were weighed with a precision of \(\pm ~0.5~\hbox {g}\). The \(^{100}\hbox {Mo}\) content was determined to range from 95.0 to 97.0% (\(\pm 0.2\%\)) by a series of ICP-MS measurements for the raw materials used for the crystals production. The systematic uncertainty for the \(^{100}\hbox {Mo}\) content corresponds to an almost negligible \(\pm ~0.057\) kg day.

Table 3 \(^{48{{\text {depl}}}}\hbox {Ca}^{100}\hbox {MoO}_{4}\) and \(^{100}\)Mo exposure of the AMoRE-Pilot experiment after each event selection method
Fig. 7
figure 7

The energy spectra gathered by \(^{48{{\text {depl}}}}\hbox {Ca}^{100}\hbox {MoO}_{4}\) detectors with the 111 kg day exposure in the ROI before and after applying coincidence tagging techniques. Each spectrum represents the result after the cut criterion overlaying the previous accumulation. The solid green line is the fitted response function describing the flat background and the excluded \(0\nu \beta \beta \) peak corresponding to the 90% confidence level (C.L.) limit of \(T_{1/2}^{0\nu } > 9.5\times 10^{22}\) year

Table 4 \(\alpha \) activity levels in the Pilot crystals. The limits are for 90% C.L

The total massic \(\alpha \) activities and the massic \(\alpha \) activities of \(^{212}\hbox {Bi}\) in the crystals are summarized in Table 4. With the exception of Crystal 2, the total alpha activity levels (of the remaining 5 crystals) are 5–18 mBq/kg. Because of the alpha rejection capability that is provided by the combination of PID parameters (see Table 2), alpha backgrounds by themselves do not affect the background level in the ROI. The only significant internal background in the crystals that could produce background signals near the 3034 keV ROI is the decay of \(^{208}\)Tl with \(Q_\beta = 4999\) keV, following the alpha decay of \(^{212}\)Bi. Our background analysis shows that the internal \(^{212}\)Bi activity limit in the 5 low-background crystals are as low as 3.1–13 \(\upmu \hbox {Bq/kg}\) (90% C.L.). Crystal 2 has an exceptionally high \(^{210}\)Po internal activity level of approximately 0.52 Bq/kg, and a \(^{212}\)Bi level that is also high, (\(60.7\pm 8.5\)\(\upmu \hbox {Bq/kg}\). The \(^{48{{\text {depl}}}}\hbox {Ca}^{100}\hbox {MoO}_{4}\) crystals for the next AMoRE phases will have \(^{212}\)Bi levels that are similar to those in the 5 low-background crystals. To further suppress the level of internal background, other types of Mo-containing crystals are also being investigated.

5 \(0\nu \beta \beta \) analysis

5.1 \(0\nu \beta \beta \) peak detection efficiency

Monte-Carlo simulations using the GEANT4 package [50] for \(0\nu \beta \beta \) events generated with the DECAY0 code [51] were carried out to evaluate the efficiency for full energy deposition in the crystals. Taking the actual dimensions of the crystals into account for the simulations, peak detection efficiencies of 79.3–82.4% in the 3024–3044 keV energy region were found for \(10^{5}\) \(0\nu \beta \beta \) events generated in each crystal, even though the masses of the lightest and heaviest ones differ by nearly a factor of two. The largest uncertainty in the peak detection efficiency estimation is due to the crystal dimensions. A simple scaling method is used to determine an uncertainty of 0.01% in absolute detection efficiency.

Table 5 Detection efficiency for the \(0\nu \beta \beta \) peak of \(^{100}\)Mo (%)
Table 6 Parameters of the response function (Eq. (8)) at the \(0\nu \beta \beta \) decay Q-value for each detector

The efficiency for selecting \(0\nu \beta \beta \) events is further adjusted to account for the cuts used for \(\beta /\gamma \) identification. We applied \(\pm 2\sigma \) cuts to select \(\beta /\gamma \)-induced events using both L/H and RT parameters. The selection efficiency for each cut is 95.45%. After consideration of the correlation between the two parameters in the ROI, the selection efficiency was determined to be (\(91.2^{+0.1}_{-0.4}\))%.

The overall detection efficiency for the present experiment is (\(74.4^{+0.1}_{-0.3}\))% and summarized in Table 5.

5.2 Model of background in the ROI

The 50 events in the ROI that survived all of the selection requirements correspond to a background level of 0.55 counts/(keV kg year) (ckky). These events were fitted with a linear background function plus a \(0\nu \beta \beta \) signal term. Using the energy calibration and the response function for a mono-energetic \(\gamma \) peak, the signal model function for each crystal is taken as the first term in Eq. (7) with parameters of \(\mu _{0\nu \beta \beta }\), \(\sigma _{0\nu \beta \beta }\), and \(\lambda _{0\nu \beta \beta }\). The uncertainty of \(\mu _{0\nu \beta \beta }\) at the Q value, denoted as \(\epsilon _{\mu _{0\nu \beta \beta }}\), reflects the uncertainty in the energy calibration. The value of the skewness parameter \(\lambda \) is scaled from the calibration data with the assumption that it is inversely proportional to the energy \(\mu \). The standard deviation \(\sigma \) of the EMG function also depends on the energy. The value of \(\sigma _{0\nu \beta \beta }\) at the Q value is determined from the \(\sigma \) values of other peaks using the relation

$$\begin{aligned} \sigma ^2(E) =\sigma _b^2 + c_1 E + c_2 E^2, \end{aligned}$$

where \(\sigma _b\) is the base resolution independent of the energy input, and \(c_2\) represents a linear relation of energy resolution with respect to the energy. The value of \(c_1\) represents an additional term arising from the non-linearity of the energy calibration and the combined effect from various other sources of \(E^2\) dependence. This \(E^2\) term is attributed to a position dependence of the signal-amplitude in the crystal, imperfect phonon collection, and temperature fluctuations. The determined values of \(\epsilon _{\mu _{0\nu \beta \beta }}\), \(\sigma _{0\nu \beta \beta }\), and \(\tau _{0\nu \beta \beta }=1/\lambda _{0\nu \beta \beta }\) are listed in Table 6. The normalized detector response function, \(f_1\), for a mono-energetic signal in the presence of a constant background is:

$$\begin{aligned} \begin{aligned}&f_{1}(E_{i,j}~|~\mu _i,\sigma _i,\lambda _i,n_{i,s},n_{i,b}) \\&\quad =\Big ( \frac{n_{i,s}}{n_{i,s}+n_{i,b}}\Big ) \cdot f_{{\text {EMG}}}(E_{i,j};\mu _i,\sigma _i,-\lambda _i)\\&\qquad +\Big (\frac{n_{i,b}}{n_{i,s}+n_{i,b}}\Big ) \cdot \frac{1}{W}~, \end{aligned} \end{aligned}$$

where the n values are the expected numbers of events, and W is the 300 keV ROI interval. The indices s and b denote the signal and the background, respectively; and the indices i and j indicate the crystal number and the event number for each crystal, respectively.

5.3 Sensitivity to \(0\nu \beta \beta \) decay of \(^{100}\)Mo

Assuming Gaussian distributions for \(\mu _i\), \(\sigma _i\) and \(\lambda _i\) described in Sect. 3, we marginalize the parameters in the likelihood function for each crystal. We then construct simultaneous unbinned extended likelihood functions for the six crystals and maximized the likelihood to fit the data. The best fit value for the decay rate \(\Gamma _{0\nu }\) is zero and there is no evidence of an \(^{100}\)Mo \(0\nu \beta \beta \) signal in the current data. We evaluate \(\Gamma ^{{\text {Limit}}}_{0\nu } < 7.3\times 10^{-24}~\hbox {year}^{-1}\) at 90% C.L. as indicated in Fig. 8. The corresponding half-life limit is:

$$\begin{aligned} T^{0\nu }_{1/2} > 9.5\times 10^{22}~{\text {year}}. \end{aligned}$$
Fig. 8
figure 8

Profile likelihood function with respect to the decay rate, \(\Gamma _{0\nu }\). The dashed red line is the limit corresponding to 90% confidence level

The solid green line in Fig. 7 is the combined response function for all six crystals with the parameters presented in Table 6 and a \(0\nu \beta \beta \) decay rate corresponding to the 90% limit of \(9.5\times 10^{22}~{\text {year}}\).

The 111 kg day (52.1 kg day of \(^{100}\)Mo) exposure and 0.55 ckky background level translates into an expected detection sensitivity for the present measurement of \(T^{0\nu }_{1/2} > 1.1\times 10^{23}\) year [52], compatible with the experimental limit reported here.

The half-life limit reported here is near the projected experimental sensitivity of \(1.1\times 10^{23}\) year, and one order of magnitude lower than the current best limit for \(^{100}\)Mo \(0\nu \beta \beta \) decay: \(1.1\times 10^{24}~\hbox {year}\) set by NEMO-3 [27], and slightly better than the LUMINEU result of \(7\times 10^{22}~\hbox {year}\) [53] based on similar low-temperature thermal detectors and obtained with a 22 kg day of \(^{100}\)Mo exposure. Note that both the AMoRE-Pilot and the LUMINEU are in their development stages with small exposures. Both experiments are continuing to take follow-up measurements with lower backgrounds and larger detector masses, and are expected to make substantial improvements in their detection sensitivities.

5.4 Neutrino mass limit

The corresponding limit on the effective Majorana neutrino mass, \(\langle m_{\beta \beta }\rangle \), is determined from Eq. (3). Using a phase space factor of \(15.92\times 10^{-15}\,\hbox {year}^{-1}\) [15] and a representative value of \(g_A = 1.27\), the corresponding limit on the effective Majorana mass is \(\langle m_{\beta \beta } \rangle ~<~1.2-2.1~\hbox {eV}\). The uncertainty of the \(\langle m_{\beta \beta } \rangle \) limit arises from the wide range of values of nuclear matrix element calculated with different techniques [54,55,56,57,58,59]. The same calculation gives a detection sensitivity limit of 1.1–1.9 eV.

6 Discussion and conclusions

In summary, we find no evidence of \(0\nu \beta \beta \) decay of \(^{100}\)Mo with a live exposure of 111 kg day using \(^{48{{\text {depl}}}}\hbox {Ca}^{100}\hbox {MoO}_{4}\) crystals. An upper limit of \(T^{0\nu }_{1/2} > 9.5\times 10^{22}~\hbox {years}\) is achieved with a background level of 0.55 counts/(keV kg year), which will be improved in succeeding measurements.

The result reported here is the first low temperature measurement using MMC sensors with \(^{48{{\text {depl}}}}\hbox {Ca}^{100}\hbox {MoO}_{4}\) crystals for \(0\nu \beta \beta \) search experiment. Although this is not the best limit on \(0\nu \beta \beta \) decay of \(^{100}\)Mo, it provides validation of the AMoRE detection concept for a competitive \(0\nu \beta \beta \) search. The background levels seen in the present study likely have external origins, and major changes are being incorporated in the shielding arrangement to reduce them. Using the levels of radioactive contaminants measured in a series of HPGe counting measurements for the materials used in the detector assembly, simulations indicate the presence of non-negligible backgrounds that originate from the materials used for sensor readout, such as pin connectors, printed circuit boards (PCBs), and Stycast-2850 epoxy. During the preparations for the sixth AMoRE-Pilot run, Polyimide-based PCBs were checked for their radioactive contamination levels and used to replace the G10 PCBs that were used in the fifth run. In addition, no pin-connectors or Stycast epoxy were used in cryostat regions below the mixing chamber plate (below the low-background shield layers of 10 cm thick lead and 6 cm thick copper). The MC simulations suggest that almost all of the nearly constant backgrounds between 4 and 8 MeV in the electron spectrum are caused by neutron captures in the cryostat and the surrounding structures. To reduce the neutron-induced background, we installed layers of neutron shields in the detector setup. The neutron shields include a layer of 4 mm boric acid placed between the outer vacuum chamber and the lead shield, 2.5 cm thick borated-polyethylene plates, and 10–30 cm thick layers of polyethylene bricks surrounding the sides, bottom and top of the detector system. Background levels that remain after these modifications are being investigated.

The detector setup with MMC readout is designed to provide fast response time for heat and light detection. The light signals for \(\alpha \) and \(\beta /\gamma \) events have fast and slow components [33]. The time response of the MMCs to the fast component is close to the intrinsic rise-time of approximately \(200~\upmu \hbox {s}\) [41], which is one to three orders of magnitude faster than those for the thermal detectors based on NTD-Ge thermistor readouts [60,61,62,63]. It is an especially important feature because of the relatively short \(2\nu \beta \beta \) half-life of \(^{100}\)Mo (\(7.1\times 10^{18}~\hbox {year}\)) [10]. As AMoRE is scaled up to larger detector masses, unresolved pileups of two random \(2\nu \beta \beta \) signals will be an irreducible major background [64, 65]. The rate of unresolved pileups from \(^{100}\)Mo decay will be approximately 0.4 events/year in a 0.4 kg \(^{48{{\text {depl}}}}\hbox {Ca}^{100}\hbox {MoO}_{4}\) with a pile-up rejection time resolution of 1 ms, which would be a non-negligible background for a larger number of crystals with a poorer time resolution. Thus, the use of MMCs as the temperature sensors has a clear advantage: better time resolution results in reduced background levels from unresolved pileup signals in rare event searches.

After the completion of the Pilot runs, AMoRE-I will be implemented with 18 crystals. These include seven additional \(^{48{{\text {depl}}}}\hbox {Ca}^{100}\hbox {MoO}_{4}\) that are already on hand. In addition, five \(^{100}\)Mo-enriched \(\hbox {Li}_{2}\hbox {MoO}_{4}\) and \(\hbox {Na}_{2}\hbox {MoO}_{4}\) crystals that are currently being fabricated. The total mass of the AMoRE-I crystals will be approximately 6 kg (3 kg of \(^{100}\)Mo). The internal backgrounds and detector performance in phonon-scintillation detection will be investigated to determine the type of crystals to be used for the AMoRE-II, the main phase of the project with a total crystal mass of 200 kg. AMoRE-II aims at improving the effective Majorana neutrino mass sensitivity to 20–50 meV. It will be installed in Yemi lab, a new 1000 m deep underground laboratory that is under construction in Korea and will be available in 2020.