Search for new phenomena in high-mass final states with a photon and a jet from $pp$ collisions at $\sqrt{s}$ = 13 TeV with the ATLAS detector

Abstract

A search is performed for new phenomena in events having a photon with high transverse momentum and a jet collected in $36.7~\text {fb}^{-1}$ of proton–proton collisions at a centre-of-mass energy of $\sqrt{s}$ = 13 TeV recorded with the ATLAS detector at the Large Hadron Collider. The invariant mass distribution of the leading photon and jet is examined to look for the resonant production of new particles or the presence of new high-mass states beyond the Standard Model. No significant deviation from the background-only hypothesis is observed and cross-section limits for generic Gaussian-shaped resonances are extracted. Excited quarks hypothesized in quark compositeness models and high-mass states predicted in quantum black hole models with extra dimensions are also examined in the analysis. The observed data exclude, at 95% confidence level, the mass range below 5.3 TeV for excited quarks and 7.1 TeV (4.4 TeV) for quantum black holes in the Arkani-Hamed–Dimopoulos–Dvali (Randall–Sundrum) model with six (one) extra dimensions.

1 Introduction

This paper reports a search for new phenomena in events with a photon and a jet produced from proton–proton ($pp$) collisions at $\sqrt{s}$ = 13 TeV, collected with the ATLAS detector at the Large Hadron Collider (LHC). Prompt photons in association with jets are copiously produced at the LHC, mainly through quark–gluon scattering ($qg \rightarrow q\gamma$). The $\gamma$ + jet(s) final state provides a sensitive probe for a class of phenomena beyond the Standard Model (SM) that could manifest themselves in the high invariant mass ($m_{\gamma j}$) region of the $\gamma$ + jet system. The search is performed by looking for localized excesses of events in the $m_{\gamma j}$ distribution with respect to the SM prediction. Two classes of benchmark signal models are considered.

The first class of benchmark models is based on a generic Gaussian-shaped mass distribution with different values of its mean and standard deviation. This provides a generic interpretation for the presence of signals with different Gaussian widths, ranging from a resonance with a width similar to the reconstructed $m_{\gamma j}$ resolution of $\sim 2$% to wide resonances with a width up to 15%. The second class of benchmark models is based on signals beyond the SM that are implemented in Monte Carlo (MC) simulation and appear as broad peaks in the $m_{\gamma j}$ spectrum. This paper considers two scenarios for physics beyond the SM: quarks as composite particles and extra spatial dimensions. In the first case, if quarks are composed of more fundamental constituents bound together by some unknown interaction, new effects should appear depending on the value of the compositeness scale $\Lambda$. In particular, if $\Lambda$ is sufficiently smaller than the centre-of-mass energy, excited quark ($q^*$) states may be produced in high-energy $pp$ collisions at the LHC  [1,2,3]. The $q^*$ production at the LHC could result in a resonant peak at the mass of the $q^*$ ($m_{q^*}$) in the $m_{\gamma j}$ distribution if the $q^*$ can decay into a photon and a quark ($qg \rightarrow q^* \rightarrow q\gamma$). In the present search, only the SM gauge interactions are considered for $q^*$ production. In the second scenario, the existence of extra spatial dimensions (EDs) is assumed to provide a solution to the hierarchy problem [4,5,6]. Certain types of ED models predict the fundamental Planck scale $M^*$ in the $4+n$ dimensions (n being the number of extra spatial dimensions) to be at the TeV scale, and thus accessible in $pp$ collisions at $\sqrt{s}$ = 13 TeV at the LHC. In such a TeV-scale $M^*$ scenario of the extra dimensions, quantum black holes (QBHs) may be produced at the LHC as a continuum above the threshold mass ($M_{\text {th}}$) and then decay into a small number of final-state particles including photon–quark/gluon pairs before they are able to thermalize [7,8,9,10]. In this case a broad resonance-like structure could be observed just above $M_{\text {th}}$ on top of the SM $m_{\gamma j}$ distribution. The $M_{\text {th}}$ value for QBH production is taken to be equal to $M^*$ while the maximum allowed QBH mass is set to either $3M^*$ or the LHC $pp$ centre-of-mass energy of 13 TeV, whichever is smaller. The upper bound on the mass ensures that the QBH production is far from the “thermal” regime, where the classical description of the black hole and its decay into high-multiplicity final states should be used. In this paper, the extra-dimensions model proposed by Arkani-Hamed, Dimopoulous and Dvali (ADD) [11] with $n=6$ flat EDs and the one by Randall and Sundrum (RS1) [12] with $n=1$ warped ED are considered.

The ATLAS and CMS experiments at the LHC have performed searches for excited quarks in the $\gamma +\text {jet}$ final state using $pp$ collision data recorded at $\sqrt{s}$ = 7 TeV [13], 8 TeV [14, 15] and 13 TeV [16]. In the ATLAS searches, limits for generic Gaussian-shaped resonances were obtained at 7, 8 and 13 TeV while a limit for QBHs in the ADD model ($n=6$) was first obtained at 8 TeV. The ATLAS search at 13 TeV with data taken in 2015 was further extended to constrain QBHs in the RS1 model ($n=1$). No significant excess of events was observed in any of these searches, and the lower mass limits of 4.4 TeV for the $q^*$ and 6.2 (3.8) TeV for QBHs in the ADD (RS1) model were set, currently representing the most stringent limits for the decay into a photon and a jet. For a Gaussian-shaped resonance a cross-section upper limit of 0.8 (1.0) fb at $\sqrt{s}$ = 13 TeV was obtained, for example, for a mass of 5 TeV and a width of 2% (15%).

The dijet resonance searches at ATLAS [17, 18] and CMS [19] using $pp$ collisions at $\sqrt{s}$ = 13 TeV also set limits on the production cross-sections of excited quarks and QBHs. The search in the $\gamma +\text {jet}$ final state presented here complements the dijet results and provides an independent check for the presence of these signals in different decay channels.

This paper presents the search based on the full 2015 and 2016 data set recorded with the ATLAS detector, corresponding to $36.7~\text {fb}^{-1}$ of $pp$ collisions at $\sqrt{s}$ = 13 TeV. The analysis strategy is unchanged from the one reported in Ref. [16], focusing on the region where the $\gamma$ + jet system has a high invariant mass.

The paper is organized as follows. In Sect. 2 a brief description of the ATLAS detector is given. Section 3 summarizes the data and simulation samples used in this study. The event selection is discussed in Sect. 4. The signal and background modelling are presented in Sect. 5 together with the signal search and limit-setting strategies. Finally the results are discussed in Sect. 6 and the conclusions are given in Sect. 7.

2 ATLAS detector

The ATLAS detector at the LHC is a multi-purpose, forward-backward symmetric detector¹ with almost full solid angle coverage, and is described in detail elsewhere [20, 21]. Most relevant for this analysis are the inner detector (ID) and the calorimeter system composed of electromagnetic (EM) and hadronic calorimeters. The ID consists of a silicon pixel detector, a silicon microstrip tracker and a transition radiation tracker, all immersed in a 2 T axial magnetic field, and provides charged-particle tracking in the range $|\eta |<2.5$. The electromagnetic calorimeter is a lead/liquid-argon (LAr) sampling calorimeter with accordion geometry. The calorimeter is divided into a barrel section covering $|\eta |<1.475$ and two endcap sections covering $1.375<|\eta |<3.2$. For $|\eta |<2.5$ it is divided into three layers in depth, which are finely segmented in $\eta$ and $\phi$. In the region $|\eta |<1.8$, an additional thin LAr presampler layer is used to correct for fluctuations in the energy losses in the material upstream of the calorimeters. The hadronic calorimeter is a sampling calorimeter composed of steel/scintillator tiles in the central region ($|\eta |<1.7$), while copper/LAr modules are used in the endcap ($1.5<|\eta |<3.2$) regions. The forward regions ($3.1<|\eta |<4.9$) are instrumented with copper/LAr and tungsten/LAr calorimeter modules optimized for electromagnetic and hadronic measurements, respectively. Surrounding the calorimeters is a muon spectrometer that includes three air-core superconducting toroidal magnets and multiple types of tracking chambers, providing precision tracking for muons within $|\eta |<2.7$ and trigger capability within $|\eta |<2.4$.

A dedicated two-level trigger system is used for the online event selection [22]. Events are selected using a first-level trigger implemented in custom electronics, which reduces the event rate to a design value of 100 kHz using a subset of the detector information. This is followed by a software-based trigger that reduces the accepted event rate to 1 kHz on average by refining the first-level trigger selection.

3 Data and Monte Carlo simulations

The data sample used in this analysis was collected from $pp$ collisions in the 2015–2016 LHC run at a centre-of-mass energy of 13 TeV, and corresponds to an integrated luminosity of $36.7\pm 1.2~\text {fb}^{-1}$. The uncertainty was derived, following a methodology similar to that detailed in Ref. [23], from a preliminary calibration of the luminosity scale using x–y beam-separation scans performed in August 2015 and May 2016. The data are required to satisfy a number of quality criteria ensuring that the relevant detectors were operational while the data were recorded.

Monte Carlo samples of simulated events are used to study the background modelling for the dominant $\gamma +\text {jet}$ processes, to optimize the selection criteria and to evaluate the acceptance and selection efficiencies for the signals considered in the search. Events from SM processes containing a photon with associated jets were simulated using the Sherpa  2.1.1 [24] event generator, requiring a photon transverse energy $E_{\text {T}} ^{\gamma }$ above 70 GeV at the generator level. The matrix elements were calculated with up to four final state partons at leading order ($\text {LO}$) in quantum chromodynamics (QCD) and merged with the parton shower  [25] using the ME+PS@LO prescription [26]. The CT10 [27] parton distribution function (PDF) set was used in conjunction with dedicated parton shower tuning developed by the Sherpa authors. A second sample of SM $\gamma +\text {jet}$ events was generated using the LO Pythia  8.186 [28] event generator with the LO NNPDF 2.3 PDFs [29] and the A14 tuning of the underlying-event parameters  [30]. The Pythia simulation includes leading-order $\gamma$ + jet events from both the direct processes (the hard subprocesses $qg \rightarrow q\gamma$ and $q\bar{q} \rightarrow g\gamma$) and bremsstrahlung photons in QCD dijet events. To estimate the systematic uncertainty associated with the background modelling, a large sample of events was generated with the next-to-leading-order (NLO) Jetphox  v1.3.1_2 [31] program. Events were generated at parton level for both the direct and fragmentation photon contributions using the NLO photon fragmentation functions [32] and the NLO NNPDF 2.3 PDFs, and setting the renormalization, factorization and fragmentation scales to the photon $E_{\text {T}} ^{\gamma }$. Jets of partons are reconstructed using the anti-$k_{t}$ algorithm [33, 34] with a radius parameter of $R=0.4$. The generated photon is required to be isolated by ensuring that the total transverse energy of partons inside a cone of size $\Delta R =0.4$ around the photon is smaller than 7.07 GeV + $0.03\times E_{\text {T}} ^{\gamma }$, equivalent² to the photon selection for the data described in Sect. 4.

Samples of excited quark events were produced using Pythia  8.186 with the LO NNPDF 2.3 PDFs and the A14 set of tuned parameters for the underlying event. The Standard Model gauge interactions and the magnetic-transition type couplings [1,2,3] to gauge bosons were considered in the production processes of the excited states of the first-generation quarks ($u^*$, $d^*$) with degenerate masses. The compositeness scale $\Lambda$ was taken to be equal to the mass $m_{q^*}$ of the excited quark, and the coefficients $f_\mathrm {s}$, f and $f'$ of magnetic-transition type couplings to the respective SU(3), SU(2) and U(1) gauge bosons were chosen to be unity. The $q^*$ samples were generated with $m_{q^*}$ values between 0.5 and 6.0 TeV in steps of 0.5 TeV.

The QBH samples were generated using the QBH 2.02 [35] event generator with the CTEQ6L1 [36] PDF set and Pythia 8.186 for the parton shower and underlying event tuned with the A14 parameter set. The $M_{\text {th}}$ values were chosen to vary between 3.0 (1.0) and 9.0 (7.0) TeV in steps of 0.5 TeV for the QBH signals in the ADD (RS1) model. All the qg, $\bar{q}g$, gg and $q\bar{q}$ processes were included in the QBH signal production while only final states with a photon and a quark/gluon were considered for the decay. All six quark flavours were included together with their anti-quark counterparts in both the production and decay processes.

Apart from the sample generated with Jetphox which is a parton-level calculator, all the simulated samples include the effects of multiple $pp$ interactions in the same and neighbouring bunch crossings (pile-up) and were processed through the ATLAS detector simulation [37] based on Geant4  [38]. Pile-up effects were emulated by overlaying simulated minimum-bias events from Pythia  8.186, generated with the A2 tune [39] for the underlying event and the MSTW2008LO PDF set  [40]. The number of overlaid minimum-bias events was adjusted to match the one observed in data. All the MC samples except for the Jetphox sample were reconstructed with the same software as that used for collision data.

4 Event selection

Photons are reconstructed from clusters of energy deposits in the EM calorimeter as described in Ref. [41]. A photon candidate is classified depending on whether the EM cluster is associated with a conversion track candidate reconstructed in the ID. If no ID track is matched, the candidate is considered as an unconverted photon. If the EM cluster is matched to either a conversion vertex formed from two tracks constrained to originate from a massless particle or a single track with its first hit after the innermost layer of the pixel detector, the candidate is considered to be a converted photon. Both the converted and unconverted photon candidates are used in the analysis. The energy of each photon candidate is corrected using MC simulation and data as described in Ref. [42]. The EM energy clusters are calibrated separately for converted and unconverted photons, based on their properties including the longitudinal shower development. The energy scale and resolution of the photon candidates after the MC-based calibration are further adjusted based on a correction derived using $Z \rightarrow e^{+} e^{-}$ events from data and MC simulation. Photon candidates are required to have $E_{\text {T}} ^{\gamma } >25$ GeV and $|\eta ^{\gamma } |<2.37$ and satisfy the “tight” identification criteria defined in Ref. [41]. Photons are identified based on the profile of the energy deposits in the first two layers of the EM calorimeter and the energy leakage into the hadronic calorimeter. To further reduce the contamination from $\pi ^{0}$ $\rightarrow \gamma \gamma$ or other neutral hadrons decaying into photons, the photon candidates are required to be isolated from other energy deposits in an event. The calorimeter isolation variable $E_{\text {T,\,iso}}$ is defined as the sum of the $E_{\mathrm {T}}$ of all positive-energy topological clusters [43] reconstructed within a cone of $\Delta R = 0.4$ around the photon direction excluding the energy deposits in an area of size $\Delta \eta \times \Delta \phi =0.125 \times 0.175$ centred on the photon cluster. The photon energy expected outside the excluded area is subtracted from the isolation energy while the contributions from pile-up and the underlying event are subtracted event by event [44]. The photon candidates are required to have $E_{\text {T,\,iso}}^{\gamma } =E_{\text {T,\,iso}}-0.022\times E_{\text {T}} ^{\gamma }$ less than 2.45 GeV. This $E_{\text {T}} ^{\gamma }$-dependent selection requirement is used to guarantee an efficiency greater than 90% for signal photons in the $E_{\text {T}} ^{\gamma }$ range relevant for this analysis. The efficiency for the signal photon selection varies from ($90\pm 1$)% to ($83\pm 1$)% for signal events with masses from 1 to 6 TeV. The dependency on the signal mass is mainly from the efficiency of the tight identification requirement while the isolation selection efficiency is approximately ($99\pm 1$)% over the full mass range.

Jets are reconstructed from topological clusters calibrated at the electromagnetic scale using the anti-$k_{t}$ algorithm with a radius parameter $R=0.4$. The jets are calibrated to the hadronic energy scale by applying corrections derived from MC simulation and in situ measurements of relative jet response obtained from Z+jets, $\gamma$+jets and multijet events at $\sqrt{s}$ = 13 TeV [45,46,47]. Jets from pile-up interactions are suppressed by applying the jet vertex tagger [48], using information about tracks associated with the hard-scatter and pile-up vertices, to jets with $p_{\text {T}} ^\text {jet} <60$ GeV and $|\eta ^{\text {jet}} |<2.4$. In order to remove jets due to calorimeter noise or non-collision backgrounds, events containing at least one jet failing to satisfy the loose quality criteria defined in Ref. [49] are discarded. Jets passing all the requirements and with $p_{\text {T}} ^\text {jet} >20$ GeV and $|\eta ^{\text {jet}} |<4.5$ are considered in the rest of the analysis. Since a photon is also reconstructed as a jet, jet candidates in a cone of $\Delta R = 0.4$ around a photon are not considered.

This analysis selects events based on a single-photon trigger requiring at least one photon candidate with $E_{\text {T}} ^{\gamma }>140$ GeV which satisfies loose identification conditions [41] based on the shower shape in the second sampling layer of the EM calorimeter and the energy leakage into the hadronic calorimeter. Selected events are required to contain at least one primary vertex with two or more tracks with $p_{\text {T}} >400$ MeV. Photon candidates are required to satisfy the “tight” identification and isolation conditions discussed above. The kinematic requirements for the highest-$E_{\text {T}}$ photon in the events are tightened to $E_{\text {T}} ^{\gamma } >150$ GeV and $|\eta ^{\gamma } |<1.37$. The $E_{\text {T}} ^{\gamma }$ requirement is used to select events with nearly full trigger efficiency [50] while the $\eta ^{\gamma }$ requirement is imposed to enhance the signal-to-background ratio. Moreover, an event is rejected if there is any jet with $p_{\text {T}} ^\text {jet} >30$ GeV within $\Delta R <0.8$ around the photon. The presence of additional tight and isolated photons with $E_{\text {T}} ^{\gamma }>150$ GeV in events is negligible for both signal and background events, and therefore allowed. The $\gamma +\text {jet}$ system is formed from the highest-$E_{\text {T}}$ photon and the highest-$p_{\text {T}}$ jet in the event. Finally, the highest-$p_{\text {T}}$ jet in the event is required to have $p_{\text {T}} ^\text {jet} >60$ GeV and the pseudorapidity difference between the photon and the jet ($\Delta \eta _{\gamma j} \equiv |\eta ^{\gamma }-\eta ^{\text {jet}} |$) must be less than 1.6 to enhance signals over the $\gamma +\text {jet}$ background, which typically has a large $\Delta \eta _{\gamma j}$ value. After applying all the selection requirements, $6.34\times 10^{5}$ events with an invariant mass ($m_{\gamma j}$) of the selected $\gamma +\text {jet}$ system greater than 500 GeV remain in the data sample.

5 Statistical analysis

The data are examined for the presence of a significant deviation from the SM prediction using a test statistic based on a profile likelihood ratio  [51]. Limits on the visible cross-section for generic Gaussian-shaped signals and limits on the cross-section times branching ratio for specific benchmark models are computed using the CL$_{\text {S}}$ prescription  [52]. The details of the signal and background modelling used for the likelihood function construction are discussed in Sects. 5.1 and 5.2 while a summary of the statistical procedures used to establish the presence of a signal or set limits on the production cross-sections for new phenomena is given in Sect. 5.3.

5.1 Signal modelling

The signal model is built starting from the probability density function (pdf), $f_{\text {sig}}(m_{\gamma j})$, of the $m_{\gamma j}$ distribution at the reconstruction level. For a Gaussian-shaped resonance with mass $m_{\text {G}}$, the $m_{\gamma j}$ pdf is modelled by a normalized Gaussian distribution with the mean located at $m_{\gamma j} = m_{\text {G}}$. The standard deviation of the Gaussian distribution is chosen to be 2, 7 or 15% of $m_{\text {G}}$, where 2% approximately corresponds to the effect of the detector resolution on the reconstruction of the photon–jet invariant mass. For the $q^*$ and QBH signals, the $m_{\gamma j}$ pdfs are created from the normalized reconstructed $m_{\gamma j}$ distributions after applying the selection requirements described in Sect. 4 using the simulated MC events, and a kernel density estimation technique [53] is applied to smooth the distributions. The signal pdfs for intermediate mass points at which signal events were not generated are obtained from the simulated signal samples by using a moment-morphing method [54]. The signal template for the $q^*$ and QBH signals is then constructed as $f_{\text {sig}}(m_{\gamma j}) \times (\sigma \cdot B \cdot A \cdot \varepsilon ) \times \mathcal {L}_{\text {int}}$, where the $f_{\text {sig}}$ is scaled by the product of the cross-section times branching ratio to a photon and a quark or gluon ($\sigma \cdot {B}$), acceptance (A), selection efficiency ($\varepsilon$) and the integrated luminosity ($\mathcal {L}_{\text {int}}$) for the data sample. The product of the acceptance times efficiency ($A \cdot \varepsilon$) is found to be about 50% for all the $q^*$ and QBH models, varying only by a few percent with $m_{q^*}$ or $M_{\text {th}}$. This dependence is accounted for in the model by interpolating between the generated mass points using a third-order spline. For the $q^*$ and QBH signals, limits are set on $\sigma \cdot {B}$ after correcting for the acceptance and efficiency $A \cdot \varepsilon$ of the selection criteria.

Experimental uncertainties in the signal yield arise from uncertainties in the luminosity ($\pm 3.2\%$), photon identification efficiency ($\pm 2\%$), trigger efficiency ($\pm 1\%$ as measured in Ref. [50] ) and pile-up dependence ($\pm 1\%$). The impact of the uncertainties in the photon isolation efficiency, photon and jet energy scales and resolutions is negligible. A 1% uncertainty in the signal yield is included to account for the statistical error in the acceptance and selection efficiency estimates due to the limited size of the MC signal samples. The impact of the PDF uncertainties on the signal acceptance is found to be negligible compared to the other uncertainties. The photon and jet energy resolution uncertainties ($\pm 2\%$ of the mass) are accounted for as a variation of the width for the Gaussian-shaped signals. The impact of the resolution uncertainty on intrinsically large width signals is found to be negligible and thus not included in the signal models for the $q^*$ and QBH. The typical difference between the peaks of the reconstructed and generator-level $m_{\gamma j}$ distributions for the excited-quark signals is well below 1%.

A summary of systematic uncertainties in the signal yield and shape included in the statistical analysis is given in Table 1.

Table 1 Summary of systematic uncertainties in the signal event yield and shape included in the fit model. The signal mass resolution uncertainty affects the generic Gaussian signal shape, while the other uncertainties affect the event yield

In order to facilitate the re-interpretation of the present results in alternative physics models, the fiducial acceptance and efficiency for events with the invariant mass of the $\gamma$ + jet system around $m_{q^*}$ or $M_{\text {th}}$ (referred to as “on-shell” events hereafter) are also provided. The chosen $m_{\gamma j}$ ranges are $0.6m_{q^*}<m_{\gamma j} <1.2m_{q^*}$ for the $q^*$ signal and $0.8M_{\text {th}}<m_{\gamma j} <3.0M_{\text {th}}$ for the QBH signal. The fiducial region at particle level, as summarized in Table 2, is chosen to be close to the one used in the event selection at reconstruction level.

Table 2 Requirements on the photon and jet at particle level to define the fiducial region and on the detector-level quantities for the selection efficiency

The fiducial acceptance $A_\mathrm {f}$, defined as the fraction of generated on-shell signal events falling into the fiducial region, increases from 56 to 63% with increasing signal mass $M_{\text {th}}$ from 1.0 to 6.5 (9.0) TeV for the QBH in the RS1 (ADD) model. The $A_\mathrm {f}$ value for the $q^*$ model varies very similarly to that for the RS1 QBH signal. The rise in the fiducial acceptance as a function of $M_{\text {th}}$ ($m_{q^*}$) is driven mainly by the increase of the efficiency for the photon $\eta$ requirement since the photons tend to be more central as $M_{\text {th}}$ ($m_{q^*}$) becomes larger.

The fiducial selection efficiency $\varepsilon _\mathrm {f}$ is defined as the ratio of the number of on-shell events in the particle-level fiducial region passing the selection at the reconstruction level, including photon identification, isolation and jet quality criteria, to the number of generated on-shell events in the particle-level fiducial region. The migration of generated on-shell events outside the particle-level fiducial region into the selected sample at the reconstruction level is found to be negligible. The fiducial selection efficiency decreases from 88 (86) to 82 (80)% within the same $M_{\text {th}}$ ranges as above for the RS1 (ADD) QBH model and is not highly dependent on the kinematics of the assumed signal production processes. The $\varepsilon _\mathrm {f}$ for the $q^*$ model behaves very similarly to that for the RS1 QBH model. The reduction in the fiducial selection efficiency is caused mainly by the inefficiency of the shower shape requirements used in the photon identification for high-$E_{\text {T}}$ photons. The fiducial acceptance and selection efficiencies for the three benchmark signal models are shown in Fig. 1 as functions of $m_{q^*}$ or $M_{\text {th}}$.

Fig. 1

a Fiducial acceptance and b selection efficiencies for the three signal models considered in the analysis as a function of the excited-quark mass $m_{q^*}$ or the QBH threshold mass $M_{\text {th}}$. The fiducial region definition is detailed in Table 2. The description of the selection criteria can be found in the text

5.2 Background modelling

The $m_{\gamma j}$ distribution of the background is modelled using a functional form. A family of functions, similar to the ones used in the previous searches for $\gamma +\text {jet}$  [13, 14, 16] and $\gamma \gamma$ resonances [55] as well as dijet resonances [17] is considered:

$$\begin{aligned} f_{\text {bg}}(x) = N (1 - x)^{p} x^{\sum _{i=0}^k a_i (\log x)^i}, \end{aligned}$$

(1)

where x is defined as $m_{\gamma j}/\sqrt{s}$, p and $a_{i}$ are free parameters, and N is a normalization factor. The number of free parameters describing the normalized mass distribution is thus $k+2$ with a fixed N, where k is the stopping point of the summation in Eq. (1). The normalization N as well as the shape parameters p and $a_{i}$ are simultaneously determined by the final fit of the signal plus background model to data. The goodness of a given functional form in describing the background is quantified based on the potential bias introduced in the fitted number of signal events. To quantify this bias the functional form under test is used to perform a signal + background fit to a large sample of background events built from the Jetphox prediction. The large Jetphox event sample is used for this purpose as the shape of the background prediction can be extracted with sufficiently small statistical uncertainty.

The parton-level Jetphox calculations do not account for effects from hadronization, the underlying event and the detector resolution. Therefore, the nominal Jetphox prediction is corrected by calculating the ratio of reconstructed jet $p_{\text {T}}$ to parton $p_{\text {T}}$ in the Sherpa $\gamma +\text {jet}$ sample and applying the parameterized ratio to the Jetphox parton $p_{\text {T}}$. In addition, an $m_{\gamma j}$-dependent correction is applied to the Jetphox prediction to account for the contribution from multijet events where one of the jets is misidentified as a photon (fake photon events). This correction is estimated from data as the inverse of the purity, defined as the fraction of real $\gamma +\text {jet}$ events in the selected sample. The purity is measured in bins of $m_{\gamma j}$ by exploiting the difference between the shapes of the $E_{\text {T,\,iso}}^{\gamma }$ distributions of real photons and jets faking photons; the latter typically have a large $E_{\text {T,\,iso}}^{\gamma }$ value due to nearby particles produced in the jet fragmentation. The purity is estimated by performing a two-component template fit to the $E_{\text {T,\,iso}}^{\gamma }$ distribution in bins of $m_{\gamma j}$. The templates of real- and fake-photon isolation distributions are obtained from MC (Sherpa) simulation and from data control samples, respectively. The $E_{\text {T,\,iso}}^{\gamma }$ variable for real photons from Sherpa simulation is corrected to account for the observed mis-modelling in the description of isolation profiles between data and MC events in a separate control sample. The template for fake photons is derived in a data sample where the photon candidate fails to satisfy the tight identification criteria but fulfils a looser set of identification criteria. Details about the correction to the real-photon template and the derivation of the fake-photon template are given in Ref. [56]. To reduce the bias in the $E_{\text {T,\,iso}}^{\gamma }$ shape due to different kinematics, both the real- and fake-photon templates are obtained by applying the same set of kinematic requirements used in the main analysis. As an example, Fig. 2 shows the $E_{\text {T,\,iso}}^{\gamma }$ distribution of events within the range $1.0<m_{\gamma j} <1.1$ TeV, superimposed on the best-fit result. This procedure is repeated in every bin of the $m_{\gamma j}$ distribution and the resulting estimate of the purity is shown as a function of $m_{\gamma j}$ in Fig. 3. The uncertainty in the measured purity includes both the statistical and systematic uncertainties. The latter are estimated by recomputing the purity using different data control samples for the fake-photon template or alternative templates for real photons obtained from Pythia simulation or removing the data-to-MC corrections applied to $E_{\text {T,\,iso}}^{\gamma }$ in the Sherpa sample and by symmetrizing the variations. The variation from different data control samples for the fake-photon template has the largest effect on the purity (4% at $1.0< m_{\gamma j} < 1.1$ TeV). The measured purity is approximately constant at 93% over the $m_{\gamma j}$ range above 500 GeV, indicating that the fake-photon contribution does not depend significantly on $m_{\gamma j}$. Figure 3 shows the $m_{\gamma j}$ distribution in data compared to the corrected Jetphox $\gamma +\text {jet}$ prediction normalized to data in the $m_{\gamma j} > 500$ GeV region.

Theoretical uncertainties in the Jetphox prediction are computed by considering the variations induced by $\pm 1\sigma$ of the NNPDF 2.3 PDF uncertainties, by switching between the nominal NNPDF 2.3 and CT10 or MSTW2008 PDFs, by the variation of the value of the strong coupling constant by $\pm 0.002$ around the nominal value of 0.118 and by the variation of the renormalization, factorization and fragmentation scales between half and twice the photon transverse momentum. The differences between data and the corrected Jetphox prediction shown in Fig. 3 are well within the uncertainties associated with the perturbative QCD prediction.

Fig. 2

Distribution of $E_{\text {T,\,iso}}^{\gamma } =E_{\text {T,\,iso}}-0.022\times E_{\text {T}} ^{\gamma }$ for the photon candidates in events with $1.0<m_{\gamma j} <1.1$ TeV, and the comparison with the result of the template fit. Real- and fake-photon components determined by the fit are shown by the green dashed and red dot-dashed histograms, respectively, and the sum of the two components is shown as the solid blue histogram. The blue band shows the systematic uncertainties in the real- plus fake-photon template. The last bin of the distribution includes overflow events. The vertical dashed line corresponds to the isolation requirement used in the analysis. The photon purity determined from the fit for the selected sample in the $1.0<m_{\gamma j} <1.1$ TeV mass range is 93%

Fig. 3

Distribution of the invariant mass of the $\gamma$ + jet system as measured in the $\gamma +\text {jet}$ data (dots), compared with the Jetphox (green histogram) $\gamma +\text {jet}$ predictions. The Jetphox distribution is obtained after correcting the parton-level spectrum for showering, hadronization and detector resolution effects as described in the text. The distributions are divided by the bin width and the Jetphox spectrum is normalized to the data in the $m_{\gamma j}$ range above 500 GeV. The ratio of the data to Jetphox prediction as a function of $m_{\gamma j}$ is shown in the middle panel (green histogram): the theoretical uncertainty is shown as a shaded band. The statistical uncertainty from the data sample and the sum of the statistical uncertainty plus the systematic uncertainty from the background subtraction are shown as inner and outer bars respectively. The measured $\gamma +\text {jet}$ purity as a function of $m_{\gamma j}$ is presented in the bottom panel (black histogram): the statistical uncertainty of the purity measurement is reported as the inner error bar while the total uncertainty is shown as the outer error bar

The number of signal events extracted by the signal + background fit to the pure background model described above is called the spurious signal  [57] and it is used to select the optimal functional form and the $m_{\gamma j}$ range of the fit. In order to account for the assumption that the corrected Jetphox prediction itself is a good representation of the data, the fit is repeated on modified samples obtained by changing the nominal shape to account for several effects: firstly, the nominal distribution is corrected to follow the envelope of the changes induced by $\pm 1\sigma$ variations of the NNPDF 2.3 PDF uncertainty, the variations between the nominal NNPDF 2.3 and CT10 or MSTW2008 PDFs, the variation of the value of the strong coupling constant by $\pm 0.002$ around the nominal value of 0.118 and the variation of the renormalization, factorization and fragmentation scales between half and twice the photon transverse momentum; secondly the corrections for the hadronization, underlying event and detector effects are removed; and finally the corrections for the photon purity are changed within their estimated uncertainty. The largest absolute fitted signal from all variations of the nominal background sample discussed above is taken to be the spurious signal.

The spurious signal is evaluated at a number of hypothetical masses over a large search range. It is required to be less than 40% of the background’s statistical uncertainty, as quantified by the statistical uncertainty of the fitted spurious signal, anywhere in the investigated search range. In this way the impact of the systematic uncertainties due to background modelling on the analysis sensitivity is expected to be subdominant with respect to the statistical uncertainty. Functional forms that cannot meet this requirement are rejected. For different signal models, the functional form and fit range are determined separately. All considered functions with k up to two (four parameters) are found to fulfil the spurious-signal requirement when fitting in the range $1.1<m_{\gamma j} <6.0$ TeV for the $q^*$ signal and $1.5~(2.5)<m_{\gamma j} <6.0~(8.0)$ TeV for the RS1 (ADD) QBH signal. To further consolidate the choice of nominal background functional form, an F test [58] is performed to determine if the change in the $\chi ^2$ value obtained by fitting the Jetphox sample with an additional parameter is significant. The test indicates that the $k=0$ (1) functional form with two (three) parameters can describe the present data sufficiently well over the entire fit range for the QBH ($q^*$) signal search, and there is no improvement by adding more parameters to the background fit function.

Given the fit range determined by the spurious signal test, the search is performed for the $q^*$ (RS1 and ADD QBH) signal within the $m_{\gamma j}$ range above 1.5 (2.0 and 3.0) TeV, to account for the width of the expected signal. The estimated spurious signal for the selected functional form is converted into a spurious-signal cross-section ($\sigma _{\text {spur}}$), which is included as the uncertainty due to background modelling in the statistical analysis. The spurious-signal cross-section, and the ratio of the spurious-signal cross-section to its uncertainty ($\delta \sigma _{\mathrm {spur}}$) and to the signal cross-section ($\sigma _{\text {model}}$) for the three benchmark models under investigation are given in Table 3 in the different search ranges. While both $\sigma _{\text {spur}}$ and $\sigma _{\text {spur}}$/$\delta \sigma _{\mathrm {spur}}$ decrease with the hypothesized signal mass, the ratio $\sigma _{\text {spur}}$/$\sigma _{\text {model}}$ increases with $m_{q^*}$ or $M_{\text {th}}$, becoming as large as 15% in the case of excited quarks with $m_{q^*}$ = 6 TeV.

Table 3 Spurious-signal cross-sections ($\sigma _{\text {spur}}$), and the ratio of the spurious-signal cross-sections to their uncertainties ($\delta \sigma _\mathrm {spur}$) and to the signal cross-sections ($\sigma _{\text {model}}$) for the three benchmark models. The values of these quantities are given at the boundaries of the search range reported in the first row

A similar test is performed to determine the functional form and fit ranges for the Gaussian-shaped signal with a 15% width. The test indicates that the same functional form and fit range as those used for the $q^*$ signal are optimal for a wide-width Gaussian signal. The same functional form and mass range is used for all the Gaussian signals.

5.3 Statistical tests

A profile-likelihood-ratio test statistic is used to quantify the compatibility between the data and the SM prediction, and to set limits on the presence of possible signal contributions in the $m_{\gamma j}$ distribution. The likelihood function $\mathcal {L}$ is built from a Poisson probability for the numbers of observed events, n, and expected events, N, in the selected sample:

$$\begin{aligned} \mathcal {L} = \text {Pois}(n|N(\varvec{\theta })) \times \left( \prod _{i=1}^n f(m_{\gamma j} ^i,\varvec{\theta })\right) \times G(\varvec{\theta }), \end{aligned}$$

where $N(\varvec{\theta })$ is the expected number of candidates, $f(m_{\gamma j} ^i, \varvec{\theta })$ is the value of the probability density function of the invariant mass distribution evaluated for each candidate event i and $\varvec{\theta }$ are nuisance parameters. The $G(\varvec{\theta })$ term collects the set of constraints on the nuisance parameters associated with the systematic uncertainties in the signal yield, in the spurious signal and in the resolution (only for Gaussian signals) and it is represented by normal distributions centred at zero and with unit variance.

The pdf of the $m_{\gamma j}$ distribution is given as the normalized sum of the signal and background pdfs:

$$\begin{aligned} f(m_{\gamma j} ^i,\varvec{\theta }) = \frac{1}{N} \left[ N_{\text {sig}} (\varvec{\theta } _{\mathrm {yield}}) f_{\text {sig}}(m_{\gamma j} ^i) + N_{\text {bg}} f_{\text {bg}}(m_{\gamma j} ^i,\varvec{\theta } _{\mathrm {bkg}})\right] , \end{aligned}$$

where $f_{\text {sig}}$ and $f_{\text {bg}}$ are the normalized signal and background $m_{\gamma j}$ distributions described in the previous sections. The $\varvec{\theta } _{\mathrm {yield}}$ are nuisance parameters associated with the signal yield uncertainties (constrained) while $\varvec{\theta } _{\mathrm {bkg}}$ are the nuisance parameters of the background shape (unconstrained). The expected number of events N is given by the sum of the expected numbers of signal events ($N_{\text {sig}}$) and background events ($N_{\text {bg}}$). The $N_{\text {sig}}$ term can be expressed as

$$\begin{aligned} N_{\text {sig}} (\varvec{\theta } _{\mathrm {yield}})= & {} N_{\text {sig}}^{\text {model}} + N_{\text {sig}}^{\text {spur}} \\= & {} (\sigma _{\text {model}} \cdot B \cdot A \cdot \varepsilon \cdot F(\varvec{\delta }_{\varepsilon },\varvec{\theta }_{\varepsilon })+ \sigma _{\text {spur}} \cdot \theta _{\text {spur}}) \nonumber \\&\times \mathcal {L}_{\text {int}} \times F(\delta _{\mathcal {L}},\theta _{\mathcal {L}}), \end{aligned}$$

where $\sigma _{\text {spur}}$ and $\theta _{\text {spur}}$ are the spurious-signal cross-section described in Sect. 5.2 and its nuisance parameter while $\mathcal {L}_{\text {int}}$ and $F(\delta _{\mathcal {L}},\theta _{\mathcal {L}})$ are the integrated luminosity and its uncertainty. Apart from the spurious signal, systematic uncertainties with an estimated size $\delta _{X}$ are incorporated into the likelihood by multiplying the relevant parameter of the statistical model by a factor $F(\delta _{X},\theta _{X})=\mathrm {e}^{\delta _{X} \theta _{X}}$. The parameter of interest in the fit to Gaussian-shaped resonances is the visible cross-section $\sigma _{\text {model}} \cdot B \cdot A \cdot \varepsilon$ while that in the fit to $q^*$ and QBH signals is $\sigma _{\text {model}} \cdot B$. For the latter case, the additional nuisance parameters for the signal efficiency uncertainties $F(\varvec{\delta }_{\varepsilon },\varvec{\theta }_{\varepsilon })$ are included.

The significance of a possible deviation from the SM prediction is estimated by computing the $p_0$ value, defined as the probability to observe, under the background model hypothesis, an excess at least as large as the one observed in data. Upper limits are set at 95% confidence level (CL) with a modified frequentist CL$_{\text {S}}$ method on the visible cross-section ($\sigma _{\text {model}} \cdot B \cdot A \cdot \varepsilon$) for the Gaussian-shaped resonances or on the signal cross-section times branching ratio ($\sigma _{\text {model}} \cdot B$) for the $q^*$ and QBH signals by identifying the value for which the CL$_{\text {S}}$ value is equal to 0.05.

6 Results

The photon–jet invariant mass distributions obtained from the selected data are shown in Fig. 4, together with the background-only fits using the model described in Sect. 5.2 and expected distributions from the signal models under test. No significant deviation from the background prediction is observed in any of the distributions. The most significant excess is observed at 1.8 TeV with the assumption of the 2%-width Gaussian model for a local significance of 2.1 standard deviations.

Fig. 4

Distributions of the invariant mass of the $\gamma$ + jet system of the observed events (dots) in $36.7~\text {fb}^{-1}$ of data at $\sqrt{s}$ = 13 TeV and fits to the data (solid lines) under the background-only hypothesis for searches in the a excited quarks, b QBH (RS1) with $n=1$ and c QBH (ADD) with $n=6$ models. The $\pm 1\sigma$ uncertainty in the background prediction originating from the uncertainties in the fit function parameter values is shown as a shaded band around the fit. The predicted signal distributions (dashed lines) for the $q^*$ model with $m_{q^*}$ = 5.5 TeV and the QBH model with $M_{\text {th}}$ = 4.5 (7.0) TeV based on RS1 (ADD) are shown on top of the background predictions. The bottom panels show the bin-by-bin significances of the data–fit differences, considering only statistical uncertainties

Limits are placed at 95% CL on the visible cross-section in the case of generic Gaussian-shaped resonances and on the production cross-section times branching ratio to a photon and a quark or gluon for the excited-quark and QBH signals. The results are shown in Fig. 5 for the Gaussian signals with the width varying between 2 and 15%, and in Fig. 6 for the benchmark signal models. The Gaussian signals are excluded for visible cross-sections above 0.25–1.1 fb (0.08–0.2 fb), depending on the width, at a mass $m_{\text {G}}$ of 3 TeV (5 TeV). In the case of the benchmark signal models considered in this analysis, the presence of a signal with a mass below 5.3, 4.4 and 7.1 TeV for the excited quarks, RS1 and ADD QBHs, can be excluded at 95% CL. The limits improve on those in Ref. [16] by about 0.9, 0.6 and 0.9 TeV for the excited quarks, RS1 and ADD QBHs, respectively.

Fig. 5

Observed (solid lines) and expected (dotted lines) 95% CL upper limits on the visible cross-sections $\sigma \cdot B \cdot A \cdot \varepsilon$ in $36.7~\text {fb}^{-1}$ of data at $\sqrt{s}$ = 13 TeV as a function of the mass $m_{\text {G}}$ of the Gaussian resonances with three different Gaussian widths between 2 and 15%. The calculation is performed using ensemble tests at mass points separated by 100 GeV over the search range

Fig. 6

Observed 95% CL upper limits (solid line with dots) on the production cross-section times branching ratio $\sigma \cdot {B}$ to a photon and a quark or gluon in $36.7~\text {fb}^{-1}$ of data at $\sqrt{s}$ = 13 TeV for the a excited-quarks, b QBH (RS1) with $n=1$ and c QBH (ADD) with $n=6$ models. The limits are placed as a function of $m_{q^*}$ for the excited quarks and $M_{\text {th}}$ for the QBH signals. The calculation is performed using ensemble tests at mass points separated by 200 (500) GeV for the RS1 (ADD) model over the search range. For the $q^*$ model the step size is 250 GeV up to 5 TeV and then 200 GeV up to 6 TeV. The limits expected if a signal is absent (dashed lines) are shown together with the $\pm 1\sigma$ and $\pm 2\sigma$ intervals represented by the green and yellow bands, respectively. The theoretical predictions of $\sigma \cdot {B}$ for the respective benchmark signals are shown by the red solid lines.

7 Conclusion

A search is performed for new phenomena in events having a photon with high transverse momentum and a jet collected in $36.7~\text {fb}^{-1}$ of $pp$ collision data at a centre-of-mass energy of $\sqrt{s}$ = 13 TeV recorded with the ATLAS detector at the LHC. The invariant mass distribution of the $\gamma$ + jet system above 1.1 TeV is used in the search for localized excesses of events. No significant deviation is found. Limits are set on the visible cross-section for generic Gaussian-shaped resonances and on the production cross-section times branching ratio for signals predicted in models of excited quarks or quantum black holes. The data exclude, at 95% CL, the mass range below 5.3 TeV for the excited quarks and 7.1 (4.4) TeV for the quantum black holes with six (one) extra dimensions in the Arkani-Hamed–Dimopoulos–Dvali (Randall–Sundrum) model. These limits supersede the previous ATLAS exclusion limits for excited quarks and quantum black holes in the $\gamma +\text {jet}$ final state.