Introduction

Photonuclear reactions can be studied in ultra-peripheral collisions (UPCs) of heavy ions where the two nuclei pass by with an impact parameter larger than the sum of their radii. Hadronic interactions are suppressed and the dominant electromagnetic interactions are mediated by photons of small virtualities. The intensity of the photon flux is growing with the squared nuclear charge of the colliding ion resulting in large cross sections for the photoproduction of vector mesons in heavy-ion collisions. The photoproduction process has a clear experimental signature: the decay products of vector mesons are the only signal in an otherwise empty detector.

The physics of vector meson photoproduction is described in [1,2,3,4]. Photoproduction of vector mesons in ion collisions can either be coherent, i.e. the photon interacts consistently with all nucleons in a nucleus, or incoherent, i.e. the photon interacts with a single nucleon. Experimentally, one can distinguish between these two production types through the typical transverse momentum of the produced vector mesons, which is inversely proportional to the transverse size of the target. While the coherent photoproduction is characterized by the production of mesons with low transverse momentum (\(\langle p_{\mathrm{T}} \rangle \sim \) 60 MeV/c), the incoherent is dominated by mesons with higher values (\(\langle p_{\mathrm{T}} \rangle \sim \) 500 MeV/c). In the first case, the nuclei usually do not dissociate, but the electromagnetic fields of ultrarelativistic heavy nuclei are strong enough to develop other independent soft electromagnetic interactions accompanying the coherent photoproduction process and resulting in the excitation of one or both of the nuclei. In the second case, the nucleus breaks up and usually emits neutrons close to the beam rapidities which can be measured in zero-degree calorimeters (ZDC) placed at long distances on both sides of the detector [5]. The incoherent photoproduction can also be accompanied by the excitation and dissociation of the target nucleon resulting in even higher transverse momenta of the produced vector mesons [6].

Coherent heavy vector meson photoproduction is of particular interest because of its connection with the gluon distribution functions (PDFs) in protons and nuclei [7]. At low Bjorken-x values, nuclear parton distribution functions are significantly suppressed in the nucleus with respect to free proton PDFs, a phenomenon known as parton shadowing [8]. Shadowing effects are usually attributed to multiple scattering and addressed in various phenomenological approaches based on elastic Glauber-like rescatterings of hadronic components of the photon, Glauber–Gribov inelastic rescatterings, and high-density QCD [9,10,11,12,13,14]. Besides, different parameterizations of nuclear partonic distributions based on fits to existing data are available [15,16,17,18], however these parameterizations are affected by large uncertainties at low Bjorken-x values due to the limited kinematic coverage of the available data samples.

Heavy vector meson photoproduction measurements provide a powerful tool to study poorly known gluon shadowing effects at low x. The scale of the four-momentum transfer of the interaction is related to the mass \(m_V\) of the vector meson as \(Q^{2}~\sim ~m^{2}_{V}/4\) corresponding to the perturbative regime in the case of heavy charmonium states. The rapidity of the coherently produced \({\mathrm{c}} {{\bar{\mathrm{c}}}}\) states is related to the Bjorken-x of the gluons as \(x~=~\left( m_V/\sqrt{s_{NN}}\right) \exp \left( \pm ~y\right) \), where the sign of the exponent reflects that each of the incoming lead nuclei may act as the photon source. The gluon shadowing factor \(R_g(x,Q^2)\), i.e. the ratio of the nuclear gluon density distribution to the gluon distribution in the proton, can be evaluated via the measurement of the nuclear suppression factor defined as the square root of the ratio of the coherent vector meson photoproduction cross section on nuclei to the photoproduction cross section in the impulse approximation that is based on the exclusive photoproduction measurements with the proton target [19, 20]. The square root in this definition is motivated by the fact that the coherent photoproduction cross section is expected to scale as the square of the gluon density in leading order pQCD.

The extraction of the nuclear suppression factor in UPC measurements is complicated by the fact that the measured vector meson cross section in UPCs is expressed as a sum of two contributions since either of the colliding ions can serve as a photon source. At forward rapidities one contribution corresponds to higher photon–nucleus energies while the other to lower energies resulting in ambiguities in the extraction of the nuclear suppression factor. The midrapidity region has the advantage that both contributions are the same and the suppression factor can be extracted unambiguously in this case. Since the contribution of the high and low energies as the rapidity changes varies across models, the measurement of the rapidity dependence of the cross section for coherent charmonium production may provide a new tool to constrain the evolution of the parton distribution in the different models.

Photoproduction cross section measurements for different quarkonium species provide an opportunity to probe gluon shadowing effects at different \(Q^2\) scales. On the other hand, the comparison of excited and ground states of charmonia can shed light on the modelling of the charmonium wave functions and help to disentangle perturbative from non-perturbative effects in the model calculations [21, 22].

Charmonium photoproduction in Pb–Pb UPCs was previously studied by the ALICE Collaboration at \(\sqrt{s_{\mathrm {NN}}}~=~2.76\) TeV [23,24,25]. The coherent \(\mathrm{J}/\psi \) photoproduction cross section was measured both at midrapidity \(|y|<0.9\) and at forward rapidity \(-3.6< y < -2.6\). In addition, the CMS Collaboration studied the coherent \(\mathrm{J}/\psi \) photoproduction accompanied by neutron emission at semi-forward rapidity \(1.8< |y| < 2.3\) at \(\sqrt{s_{\mathrm {NN}}}~=~2.76\) TeV [26]. The results were compared with various models and the best description was found amongst those introducing moderate gluon shadowing in the nucleus. The ALICE measurements were used in Ref. [19] to extract the nuclear gluon shadowing factor \(R_g\) yielding \(R_g(x \sim 10^{-3}) = 0.61^{+0.05}_{-0.04}\) and \(R_g(x \sim 10^{-2}) = 0.74^{+0.11}_{-0.12}\) at the scale of the charm quark mass. The ALICE measurement of \({\uppsi '}\) photoproduction at midrapidity also supports the moderate-shadowing scenario [25]. A complementary rapidity-differential measurement of the coherent \(\mathrm{J}/\psi \) and \({\uppsi '}\) photoproduction at forward rapidity in Pb–Pb UPCs at \(\sqrt{s_{\mathrm {NN}}}~=~5.02\) TeV by the ALICE Collaboration further underlines the importance of gluon shadowing effects [27]. The gluon shadowing factor \(R_g(x \sim 10^{-2}) \sim 0.8\) was obtained under assumption that the contribution from high photon-nucleus energies, i.e. low Bjorken \(x\sim 10^{-5}\), can be neglected in the measured cross sections.

In this publication, we present the first measurement of the coherent \(\mathrm{J}/\psi \) and \({\uppsi '}\) photoproduction cross sections at the midrapidity range \(|y|<0.8\) in the Pb–Pb UPCs at \(\sqrt{s_{\mathrm {NN}}}~=~5.02\) TeV, recorded by ALICE in 2018. The \(\mathrm{J}/\psi \) photoproduction cross section in this measurement is sensitive to \(x\in (0.3,1.4)\times 10^{-3}\), a factor 2 smaller than in the previous midrapidity measurement at \(\sqrt{s_{\mathrm {NN}}}~=~2.76\) TeV [24]. This data sample is approximately 10 times larger than Pb–Pb sample at \(\sqrt{s_{\mathrm {NN}}}~=~2.76\) TeV used for the ALICE results reported in Refs. [24, 25]. The larger data sample allows for a measurement of the \(\mathrm{J}/\psi \) cross section in three rapidity intervals (\(|y|<0.15\), \(0.15<|y|<0.35\), \(0.35<|y|<0.8\)) extending the previous rapidity-differential cross section measurement in the forward range at \(\sqrt{s_{\mathrm {NN}}}~=~5.02\) TeV [27]. \(\mathrm{J}/\psi \) decays to \(\mu ^{+} \mu ^{-}\), \(e^{+} e^{-}\) and \({\mathrm{p}} {\overline{{\mathrm{p}}}}\) and \({\uppsi '}\) decays to \(\mu ^{+} \mu ^{-} \pi ^{+} \pi ^{-}\), \(e^{+} e^{-} \pi ^{+} \pi ^{-}\) and \(l^{+} l^{-}\) are investigated. The coherent \(\mathrm{J}/\psi \) production in the \({\mathrm{p}} {\overline{{\mathrm{p}}}}\) channel in UPCs is measured for the first time. The ratio of the \({\uppsi '}\) and \(\mathrm{J}/\psi \) cross sections is also measured and compared with earlier ALICE measurements [25, 27]. The measured cross sections are compared to models assuming no gluon shadowing as well as to predictions that employ moderate gluon shadowing. Shadowing models are based on a parametrization of previously available data, the leading twist approximation and several variations of the color dipole approach.

Detector description

The ALICE detector and its performance are described in [28, 29]. The main components of the ALICE detector are a central barrel placed in a large solenoid magnet (\(B=0.5\) T), covering the central pseudorapidity region, and a muon spectrometer at forward rapidity, covering the range \(-4.0<\eta <-2.5\). Three central barrel detectors, the Inner Tracking System (ITS), the Time Projection Chamber (TPC), and the Time-of-Flight detector (TOF), are used in this analysis.

The ITS is made of six silicon layers and is used for particle tracking and interaction vertex reconstruction [30]. The Silicon Pixel Detector (SPD) makes up the two innermost layers of the ITS with about \(10^7\) pixels covering the pseudorapidity intervals \(|\eta | < 2\) and \(|\eta | < 1.4\) for the inner (radius 3.9 cm) and outer (radius 7.6 cm) layers, respectively. The SPD is read out by 400 (800) chips in the inner (outer) layer with each of the readout chips also providing a trigger signal if at least one of its pixels is fired. When projected into the transverse plane, the chips are arranged in 20 (40) azimuthal regions in the inner (outer) layer allowing for a topological selection of events at the trigger level.

The TPC is used for tracking and for particle identification [31]. A 100 kV central electrode separates the two drift volumes, providing an electric field for electron drift. The two end-plates, at \(|z| = 250\) cm, are instrumented with Multi-Wire-Proportional-Chambers (MWPCs) with 560,000 readout pads, allowing high precision track measurements in the transverse plane. The z coordinate is given by the time of drift in the TPC electric field. The TPC acceptance covers the pseudorapidity region \(|\eta | <0.9\). Ionization measurements of individual track clusters are used for particle identification.

The TOF detector is a large cylindrical barrel of multigap resistive plate chambers with about 150,000 readout channels surrounding the TPC and providing very high precision timing measurement [32]. The TOF pseudorapidity coverage is \(|\eta | <0.8\). In combination with the tracking system, the TOF detector is used for charged particle identification up to a momentum of about 2.5 GeV/c for pions and kaons and up to 4 GeV/c for protons. The TOF readout channels are grouped into 1608 trigger channels (maxipads) arranged into 18 azimuthal regions and provide topological-trigger decisions.

The measurement also makes use of the three forward detectors. The V0 counters consist of two arrays of 32 scintillator tiles each, covering the interval \(2.8\mathrm<\eta <5.1\) (V0A) and \(-3.7\mathrm<\eta <-1.7\) (V0C) and positioned respectively at \(z=340\) cm and \(z=-90\) cm from the interaction point [33]. The ALICE Diffractive (AD) detector consists of two arrays of 8 scintillator tiles each arranged in two layers, covering the range \(4.9\mathrm<\eta <6.3\) (ADA) and \(-7.0\mathrm<\eta <-4.8\) (ADC) and positioned at \(z=17\) m and \(z=-19.5\) m from the interaction point, respectively [34]. Both V0 and AD can be used to veto hadronic interactions at the trigger level.

Finally, two zero-degree calorimeters ZNA and ZNC, located at \(\pm 112.5\) m from the interaction point, are used for the measurement of neutrons at beam rapidity. They have good efficiency (\(\approx 0.94\)) to detect neutrons with \(|\eta |>8.8\) and have a relative energy resolution of about 20% for single neutrons, which allows for a clear separation of events with either zero or a few neutrons at beam rapidity [5].

Data analysis

Event selection

The data analysis in this paper is based on the event sample recorded during the Pb–Pb  at \(\sqrt{s_{\mathrm {NN}}}~=~5.02\) TeV data taking period in 2018. A dedicated central barrel UPC trigger consists of topological trigger formed by at least two and up to six TOF maxipads with at least one pair of maxipads having an opening angle in azimuth larger than 150 degrees and a topological trigger formed by at least four triggered SPD chips. The triggered SPD chips are required to form two pairs, each pair with two chips in different SPD layers falling in compatible azimuthal regions. The two pairs of chips are required to have an opening angle in azimuth larger than 153 degrees. It is further vetoed by any activity within the time windows for nominal beam–beam interactions on the V0 and AD detectors on both sides of the interaction point.

The used data sample corresponds to an integrated luminosity of \(233\, \mu \mathrm{b}^{-1}\), derived from the counts of two independent reference triggers, one was based on multiplicity selection in the V0 detector and another one based on neutron detection in the ZDC. The reference trigger cross sections were determined from van der Meer scans; this procedure has an uncertainty of 2.2% [35].The determination of the live-time of the UPC trigger has an additional uncertainty of 1.5%. The total relative systematic uncertainty of the integrated luminosity is thus 2.7%. Alternatively the reference cross section can be calibrated using Glauber calculation [27]. The difference between the two methods is smaller than the quoted systematic uncertainty. Alternatively the reference cross section can be calibrated using Glauber calculation [27]. The difference between the two methods is smaller than the quoted systematic uncertainty.

Additional offline vetoes are applied on the AD and V0 detector signals to ensure the exclusive production of the charmonia. The offline selection in these detectors is more precise than vetoes at the trigger level, because it relies on larger time windows than the trigger electronics and on a more refined algorithm to quantify the signal.

Online and offline V0 and AD veto requirements may result in significant inefficiencies (denoted as veto inefficiencies) in selecting signal events with exclusive charmonium production due to additional activity induced by hadronic or electromagnetic pile–up processes from independent Pb–Pb collisions accompanying the coherent charmonium photoproduction. The probability of hadronic pile–up in the collected sample does not exceed 0.2%, however there is a significant pile–up contribution from the electromagnetic electron-pair production process. The veto inefficiency induced by these pile–up effects in the V0 and AD detectors is estimated using events selected with an unbiased trigger based only on the timing of bunches crossing the interaction region. The average veto efficiency \(\varepsilon _{\mathrm{veto}}^{\mathrm{pileup}}=0.920 \pm 0.002\) is applied to raw charmonium yields to account for hadronic and electromagnetic pile-up processes. In the forward rapidity analysis a higher veto efficiency was obtained due to missing veto on V0C detector.

Signal events with exclusive charmonium production, accompanied by electromagnetic nuclear dissociation (EMD), can be rejected if, in addition to the forward neutrons, other particles, produced at large rapidities, leave a signal either in the AD or the V0 detectors. These extra particles may come from multifragmentation or pion production processes, and the corresponding cross sections are expected to be large [36]. The amount of good events with neutrons, which are lost due to AD and V0 vetoes, is estimated using control triggers without veto on AD and/or VZERO detectors. The fraction of losses for this category of events (EMD) amounts to \(26\% \pm 4\%\) for events with a signal either in ZNA or ZNC and reach \(43\% \pm 5\%\) for events with a signal in both ZNA and ZNC. The average event loss is computed using fractions of events with and without neutrons on either side. The average veto efficiency correction \(\varepsilon _{\mathrm{veto}}^{\mathrm{EMD}} = 0.92 \pm 0.02\) is applied to raw charmonium yields to account for the EMD process.

The selected events are required to have a reconstructed primary vertex determined using at least two reconstructed tracks and having a longitudinal position within 15 cm of either side of the nominal interaction point. The analysis is aimed at the reconstruction of \(\mathrm{J}/\psi \) decaying to \(\mu ^{+} \mu ^{-}\), \(e^{+} e^{-}\), \({\mathrm{p}} {\overline{{\mathrm{p}}}}\)  and of \({\uppsi '}\) decaying to \(l^{+} l^{-}\)and \(\mathrm{J}/\psi \) \(\pi ^+\pi ^-\) followed by \(\mathrm{J}/\psi \) \(\rightarrow l^+ l^-\). Therefore, events with two or four tracks in the central barrel are required.

Two types of tracks are considered in the analysis: global tracks and ITS standalone tracks. Global tracks are reconstructed using combined tracking in ITS and TPC detectors. Tracks are required to cross at least 70 (out of 159) TPC pad-rows and to have a cluster on each of the two layers of the SPD. Each track must have a distance of closest approach to the primary vertex of less than 2 cm in the direction of z-axis. ITS standalone tracks are reconstructed using ITS clusters not attached to any global track, requiring at least four clusters in the ITS, out of which two must be in the SPD.

The two-body decays are selected by looking for events with exactly two global tracks with opposite electric charge (unlike-sign). The probability to find extra global tracks not passing the standard track selection criteria or being reconstructed only in ITS is found to be negligible. The four-body decays of \({\uppsi '}\) are selected by looking for exactly four tracks with at least two being global tracks. The kinematics of the \(\psi ' \rightarrow l^+l^-\pi ^+\pi ^-\) decay is such that pions and leptons are well separated: leptons have high \(p_{\mathrm{T}} \approx 1\) GeV/c while pions are much softer with \(p_{\mathrm{T}} \approx 0.3\) GeV/c. This feature is used to identify the pion pair. Tracks are sorted according to their \(p_{\mathrm{T}}\) and the two with lowest \(p_{\mathrm{T}}\) are assumed to be pions, while the other two are assumed to be leptons. The tagged pions and lepton pairs are required to consist of opposite-sign tracks.

To separate the \(\mathrm{J}/\psi \) \(\rightarrow \) \(\mu ^{+} \mu ^{-}\), \(e^{+} e^{-}\) and \({\mathrm{p}} {\overline{{\mathrm{p}}}}\) decays, the particle identification (PID) capabilities of the TPC and TOF detectors are used. The momenta of the tracks from \(\mathrm{J}/\psi \) decays are \(p \in (1.0, 2.0)\) GeV/c for the \(\mu ^{+} \mu ^{-}\) and \(e^{+} e^{-}\) channels and \( p \in (0.75, 1.75)\) GeV/c for the \({\mathrm{p}} {\overline{{\mathrm{p}}}}\) channel. The PID resolution of the TPC allows for complete separation of electrons and muons in the momentum range mentioned above. Since the specific ionization energy loss (\({\mathrm{d}}E/{\mathrm{d}}x\)) of electron and proton become equal at momenta around 1 GeV/c, the TPC PID is not applicable for the identification of protons from coherently produced \(\mathrm{J}/\psi \). However, the PID capabilities of the TOF detector allow for the separation of protons from other particle species in the momentum range relevant for this analysis. For the \(\mathrm{J}/\psi \) \(\rightarrow \) \({\mathrm{p}} {\overline{{\mathrm{p}}}}\) channel, at least one track is required to have valid TOF PID information. If no TOF PID is available for the second track, TPC PID is used. The \({\mathrm{d}}E/{\mathrm{d}}x\) in TPC or the Lorentz Beta factor (\(\beta = v/c\)) of each reconstructed track in TOF is measured in units of the standard deviation (\(\sigma \)) with respect to expected values for \(\mu ,e,{\mathrm{p}}\) at the given measured momentum. The track pair is accepted if \(n^{2}_{\sigma +} + n^{2}_{\sigma - } < 16\).

The charmonium photoproduction may be accompanied by pile-up from electromagnetic electron-pair production or by noise in the SPD resulting in extra fired SPD trigger chips satisfying the SPD trigger selection topology. In order to exclude contamination of events not triggered by the charmonium decay products, the fired SPD trigger chips are required to match SPD clusters corresponding to the selected tracks. It is found that 11% (7%) of the events with a \(\mathrm{J}/\psi \) candidate decaying into dimuons (di-electrons) with 4 SPD clusters cannot be matched to the fired trigger chips. The matching requirement has a much stronger effect for the 4-track decay channels of \({\uppsi '}\) removing 40% and 22% of the candidates in the \({\uppsi '}\) \(\rightarrow \) \(\mu ^{+} \mu ^{-} \pi ^{+} \pi ^{-}\) and \({\uppsi '}\) \(\rightarrow \) \(e^{+} e^{-} \pi ^{+} \pi ^{-}\)channel, respectively.

Acceptance and efficiency correction

The product of acceptance and efficiency of the \(\mathrm{J}/\psi \) and \({\uppsi '}\) reconstruction (\(\varepsilon \)) is evaluated using a large Monte Carlo (MC) sample of coherent and incoherent \(\mathrm{J}/\psi \) and \({\uppsi '}\) events generated by STARlight 2.2.0 [37] with decay particles tracked in a model of the experimental apparatus implemented in GEANT 3.21 [38]. The model includes a realistic description of the detector status during data taking and its variation with time.

For this analysis, the primary \(\mathrm{J}/\psi \) and \({\uppsi '}\) vector mesons produced in UPCs are considered to be transversely polarized. This is consistent with expectations from helicity conservation in photo production and consistent with H1 and ZEUS measurements [39,40,41]. As observed in previous experiments, both \(\mathrm{J}/\psi \) and the two pions from \({\uppsi '}\) decay are in the S-wave state resulting into the full transfer of the \({\uppsi '}\) polarisation to the \(\mathrm{J}/\psi \)[42]. The expected polarization states of primary \(\mathrm{J}/\psi \) and \({\uppsi '}\) as well as of secondary \(\mathrm{J}/\psi \) from \({\uppsi '}\) decays are properly taken into account in the MC simulations used in this analysis. These MC simulations are also used in the evaluation of the feed-down contribution to the two-body decay channels from the \(\psi ' \rightarrow {\mathrm{J}}/\psi + \pi ^{+} \pi ^{-}\) and \(\psi ' \rightarrow {\mathrm{J}}/\psi + \pi ^{0} \pi ^{0}\) decays and for modeling the signal shape and different background contributions.

The efficiency of the SPD trigger chips is measured with a data-driven approach using a minimum bias trigger. Tracks selected without requiring hits in both SPD layers are matched to the trigger chips they cross. The obtained efficiency maps are introduced on an event-by-event basis to the MC simulations. The single chip efficiency is about 92%. As the trigger requires 4 chips, the overall effect corresponds to an efficiency of about 0.72 ± 0.01.

The TOF trigger efficiency is also estimated with a data-driven approach and is taken into account in the MC simulations. The average coverage of the active TOF trigger channels is approximately 90%. The average trigger efficiency of the active channels is defined as the probability to find signals in maxipads crossed by extrapolated TPC tracks from minimum bias events and is found to be 97–98%, depending on track arrival times. More than 93% of the active channels are almost 100% efficient. The low efficiency in some channels is caused by timing alignment issues and partially disconnected or broken equipment.

Signal extraction

The extraction of coherent \(\mathrm{J}/\psi \) and \({\uppsi '}\) yields in all decay channels is performed in the rapidity interval \(|y|<0.8\). In addition, the \(\mathrm{J}/\psi \) measurements in the dielectron and dimuon channels are performed in three rapidity intervals: \(|y| < 0.15\), \(0.15< |y| < 0.35\), and \(0.35< |y| < 0.8\) where the y ranges were chosen to have approximately the same number of candidates per range. An enriched sample of coherent \(\mathrm{J}/\psi \) and \({\uppsi '}\) candidates is obtained by selecting the reconstructed candidates with transverse momentum \(p_{\mathrm{T}} < 0.2\) GeV/c.

The invariant mass distributions for dimuon and dielectron pairs reconstructed in the full rapidity range are shown in Fig. 1, left. The inclusive \(\mathrm{J}/\psi \) yields are obtained by fitting the invariant mass distributions with an exponential function describing the underlying continuum and two Crystal Ball functions to describe the \(\mathrm{J}/\psi \) and \({\uppsi '}\) signals. The \(\mathrm{J}/\psi \) pole mass and width were left free, while the tail parameters (\(\alpha \) and n) in the Crystal Ball function were fixed to the values obtained in MC simulations in order to gain higher stability of the fits. In the case of the \({\uppsi '}\) signal, all the Crystal Ball parameters were fixed to the values obtained in MC simulations.

The raw inclusive \(\mathrm{J}/\psi \) yields obtained from invariant mass fits contain contributions from the coherent and incoherent \(\mathrm{J}/\psi \) photoproduction that can be separated via the analysis of the transverse momentum spectra. The inclusive \(p_{\mathrm{T}}\)  distributions for \(\mu ^{+} \mu ^{-}\) and \(e^{+} e^{-}\) candidates around the \(\mathrm{J}/\psi \) mass are shown in the right panels of Fig. 1. These distributions are fitted with MC templates produced using STARlight, followed by full detector simulation and reconstruction, corresponding to different production mechanisms: coherent and incoherent \(\mathrm{J}/\psi \), feed-down \(\mathrm{J}/\psi \) from decays of coherent and incoherent \({\uppsi '}\) and the dilepton continuum from the \(\gamma \gamma \rightarrow l^{+}l^{-}\) process. Incoherent \(\mathrm{J}/\psi \) production with nucleon dissociation (or dissociative \(\mathrm{J}/\psi \)) is also taken into account to describe the high-\(p_{\mathrm{T}}\)  tail with the template based on the H1 parametrization [39]. Normalization of feed-down \(\mathrm{J}/\psi \) from coherent and incoherent \({\uppsi '}\) decays is constrained to the normalization of primary \(\mathrm{J}/\psi \) templates according to the feed-down fractions extracted as described below. The normalization of the dilepton continuum from the \(\gamma \gamma \rightarrow l^{+}l^{-}\) process is fixed by the results for the background description of the invariant mass fits. The combinatorial background, estimated by considering the distribution of like-sign candidates, is found to be negligible in the \(\mathrm{J}/\psi \) mass region.

The templates are fitted to the data leaving the normalization free for coherent \(\mathrm{J}/\psi \), incoherent \(\mathrm{J}/\psi \) and dissociative \(\mathrm{J}/\psi \) production. The extracted incoherent \(\mathrm{J}/\psi \) fraction \(f_{\mathrm{I}} = \frac{N^{\mathrm{incoh}}}{N^{\mathrm{coh}}}\) for \(p_{\mathrm{T}} <0.2\) GeV/c is \(4.7 \pm 0.3\%\) \((5.0 \pm 0.5)\%\) for the \(\mu ^{+} \mu ^{-}\)(\(e^{+} e^{-}\)) decay channel. The quoted fractions include the contribution of incoherent \(\mathrm{J}/\psi \) with nucleon dissociation.

The invariant mass and the \(p_{\mathrm{T}}\)  distributions for the \({\mathrm{J}}/\psi \rightarrow {\mathrm{p}}{\bar{\mathrm{p}}}\) decay channel are shown in Fig. 2. The data sample obtained in this channel is too small to fit the \(p_{\mathrm{T}}\)  distribution with MC templates. However, since the difference in resolution of \(p_{\mathrm{T}}\)  shapes of the coherent or incoherent MC samples for the \({\mathrm{p}} {\overline{{\mathrm{p}}}}\) and \(\mu ^{+} \mu ^{-}\) channels is negligible, one can expect the \(f_{\mathrm{I}}\) fraction to be the same. This is due to the fact that neither the \(\mu ^{+} \mu ^{-}\) nor the \({\mathrm{p}} {\overline{{\mathrm{p}}}}\) channels suffer from bremsstrahlung. This is not the case for dielectrons where bremsstrahlung induces the large difference in the mass and momentum resolution which affect the templates and consequently the \(f_{\mathrm{I}}\) fraction.

Fig. 1
figure 1

Left: Invariant mass distribution of \(l^{+} l^{-}\) pairs. The dashed green line corresponds to the background. The solid magenta and red lines correspond to Crystal Ball functions representing the \(\mathrm{J}/\psi \) and \({\uppsi '}\) signal, respectively. The solid blue line corresponds to the sum of background and signal functions. Right: Transverse momentum distribution of \(\mathrm{J}/\psi \) candidates in the range quoted in the figure (around the \(\mathrm{J}/\psi \) nominal mass)

Fig. 2
figure 2

Left: Invariant mass distribution of \({\mathrm{p}} {\overline{{\mathrm{p}}}}\) pairs. The dashed green line corresponds to the background description. The solid magenta and red lines correspond to Crystal Ball functions representing the \(\mathrm{J}/\psi \) and \({\uppsi '}\) signals, respectively. The solid blue line corresponds to the sum of background and signal functions. Right: Transverse momentum distribution of \(\mathrm{J}/\psi \) candidates in the range quoted in the figure (around the \(\mathrm{J}/\psi \) nominal mass)

Fig. 3
figure 3

Left: Invariant mass distribution of \(l^{+} l^{-}\) pairs. The dashed green line corresponds to the background description. The solid magenta and red line correspond to Crystal Ball functions representing the \(\mathrm{J}/\psi \) and \({\uppsi '}\) signals, respectively. The solid blue line corresponds to the sum of background and signal functions. Right: Transverse momentum distribution of \({\uppsi '}\) candidates in the mass range quoted in the figure (around the \({\uppsi '}\) mass)

Fig. 4
figure 4

Left: Invariant mass distribution for \({\uppsi '}\) \(\rightarrow \) \(\mu ^{+} \mu ^{-} \pi ^{+} \pi ^{-}\) (upper panel) and \({\uppsi '}\) \(\rightarrow \) \(e^{+} e^{-} \pi ^{+} \pi ^{-}\) (bottom panel). The green line shows the wrong sign four-track events. The red line shows the \({\uppsi '}\) signal as described in the text. Right: Transverse momentum distribution of \({\uppsi '}\) candidates in the mass range quoted in the figure (around the \({\uppsi '}\) nominal mass)

As one can see in Fig. 1, the \({\uppsi '}\) yields in the \(\mu ^{+} \mu ^{-}\) and \(e^{+} e^{-}\) channels are small and lying on top of a significant background. In order to increase the significance of the \({\uppsi '}\) signal and to reduce the statistical uncertainty, the \({\uppsi '}\) yield is extracted from the merged \(l^{+} l^{-}\) sample with significance \(\approx 3\). Figure 3 shows the merged dilepton mass spectrum together with the \(p_{\mathrm{T}}\) distribution of the dilepton candidates in the invariant mass range under the \({\uppsi '}\) mass peak. The fit to the invariant mass distribution is performed in the same way as described before.

Table 1 Raw \(\mathrm{J}/\psi \) yields, \(\varepsilon \), \(f_{\mathrm{D}}\) and \(f_{\mathrm{I}}\) fractions and coherent \(\mathrm{J}/\psi \) cross sections per decay channel