The possibility of search for new physics in ultraperipheral heavy ion collisions (UPCs) has been largely discussed in recent years (for recent reviews see, e.g. Refs. [1, 2]). Such studies are strongly motivated by the large photon–photon luminosity (\(\propto Z_1^2 Z_2^2\), where \(Z_i\) are the atomic numbers of the incident particles) and the clean environment present in UPCs [3,4,5,6,7,8,9,10,11], which enhance the production cross sections and reduce the backgrounds, allowing to derive stringent limits on some Beyond Standard Model (BSM) scenarios that predict new particles that couple predominantly to photons. In particular, the recent results reported by the ATLAS and CMS Collaborations for the diphoton production in UPCs [12, 13] provide the strongest limits to date in the properties of the axionlike particles (ALPs) in the mass region 5 GeV \(\le m_a \le \) 100 GeV. Such pseudo-scalar particles are pseudo-Nambu–Goldstone bosons predicted to occur in many extensions of the Standard Model (SM), as e.g. supersymmetry or string theories, which can be produced at colliders and decay into photons, charged leptons, light hadrons or jets, depending on the ALP mass and coupling structure (see e.g. Ref. [14]). The coupling of the pseudoscalar ALP a to photons is usually described by a Lagrangian of the form

$$\begin{aligned} \mathcal {L}=\frac{1}{2}\partial ^\mu a \partial _\mu a -\frac{1}{2}m_a^2 a^2 -\frac{1}{4}g_{a\gamma } a F^{\mu \nu }\tilde{F}_{\mu \nu }\;, \end{aligned}$$
(1)

where \(m_a\) is the ALP mass, \(g_{a\gamma }\) is the coupling constant to photons and \(\tilde{F}^{\mu \nu } = \frac{1}{2}\epsilon ^{\mu \nu \alpha \beta } F_{\alpha \beta }\), which implies that ALPs can be produced by the photon–photon fusion and can decay into a diphoton system, as represented in Fig. 1a, where the incident particles are assumed as source of photons. The presence of an axionlike particle as a resonance in the \(\gamma \gamma \rightarrow a \rightarrow \gamma \gamma \) subprocess is expect to modify the scattering rates of the light-by-light (LbL) scattering, characterized by the \(\gamma \gamma \rightarrow \gamma \gamma \) subprocess and represented in Fig. 1b, with a larger impact on the invariant mass distribution of the photon pair system. Such aspect was exploited in the studies performed in Refs. [15,16,17] as well as in the experimental analysis at the LHC [12, 13]. In particular, in the exploratory study performed in Ref. [17], we have demonstrated that a forward detector, as the LHCb, is ideal to probe an ALP with small mass (\(m_a \lesssim 10\) GeV), which is a mass region of difficult experimental access for the ATLAS and CMS detectors due to restrictions on the photon reconstructive capabilities. In Ref. [17] we have restricted our analysis for PbPb collisions and studied the ALP production assuming four different combinations for the ALP mass and coupling. One of the goals of this letter is to extend the analysis for ultraperipheral Pbp collisions at \(\sqrt{s_{_{\textsc {nn}}}}= 8.1\) TeV. Moreover, we complement the previous study for PbPb collisions by presenting the results for a larger number of combinations of ALP mass and coupling, with a special focus on the low mass region \(m_a \lesssim 5\) GeV and forward rapidities, which can be reached by the LHCb detector. For completeness, we also will present the predictions considering that the diphoton system is produced at central rapidities and that the low invariant masses range could be reached by a central detector in the forthcoming years. Another goal is to derive, for the first time, the exclusion limits on the ALP parameter space (\(m_a, \, g_{a\gamma }\)) that can be established by studying the production of diphotons with small invariant masses in ultraperipheral Pbp and PbPb collisions at the LHC. In our analysis, we will assume that the main backgrounds for the ALP production come from the LbL scattering (Fig. 1b) and the dielectron production (Fig. 1b), where both the electron and positron are misidentified as photons. As demonstrated in Refs. [17,18,19], the background associated to the diphoton production in gluon-induced processes is negligible when the exclusivity cuts are taken into account. The background associated to photoproduction of pseudoscalar resonances and \(\pi ^0\)-pairs was addressed in Ref. [20] and its contribution can, in principle, be strongly reduced through the inclusion of additional cuts on the transverse momenta of the two photons. In our study, the predictions for the signal and backgrounds will be derived using the superchic4 Monte Carlo event generator [21] including realistic cuts on rapidity, invariant mass, transverse momentum and acoplanarity of the diphoton system.

Fig. 1
figure 1

Diphoton production in Pbp/PbPb collisions by a the \(\gamma \gamma \rightarrow a \rightarrow \gamma \gamma \) subprocess and b light-by-light scattering. The dielectron production, which is a important background when the electrons are misidentified as photons, is represented in the diagram c

In the equivalent photon approximation [22], the total cross section for the hadronic process can be factorized in terms of the equivalent photon spectrum of the incident nuclei and the elementary cross section for the \(\gamma \gamma \rightarrow a \rightarrow \gamma \gamma \) process as follows

$$\begin{aligned}&\sigma \left( h_1 h_2 \rightarrow h_1 \otimes \gamma \gamma \otimes h_2;\sqrt{s_{_{\textsc {nn}}}}\right) \nonumber \\&\quad = \int \text{ d}^{2} {{\mathbf {r}}_{1}} \text{ d}^{2} {{\mathbf {r}}_{2}} \text{ d }W \text{ d }y \frac{W}{2} \, \hat{\sigma }\left( \gamma \gamma \rightarrow a \rightarrow \gamma \gamma ; W \right) N_{h_1}\nonumber \\&\qquad \times \left( \omega _{1},{{\mathbf {r}}_{1}} \right) N_{h_2} \left( \omega _{2},{{\mathbf {r}}_{2}} \right) S^2_{abs}({{\mathbf {b}}}), \end{aligned}$$
(2)

where \(h_i = p\) or Pb, \(\sqrt{s_{_{\textsc {nn}}}}\) is center-of-mass energy of the \(h_1h_2\) collision, \(\otimes \) characterizes a rapidity gap in the final state, \(\omega _i\) are the energies of the incident photons and \(W = \sqrt{4 \omega _1 \omega _2}\) is the \(\gamma \gamma \) center-of-mass energy, which is equal to the invariant mass (\(m_{\gamma \gamma }\)) of the produced \(\gamma \gamma \) system at the rapidity y. Moreover, \(N_{h_i}(\omega _i, {{\mathbf {r}}}_i)\) is the equivalent photon spectrum associated to hadron \(h_i\), which allows to estimate the number the photons with energy \(\omega _i\) at a transverse distance \({{\mathbf {r}}}_i\) from the center of hadron, defined in the plane transverse to the trajectory, which is determined by the charge form factor of the particle. The absorptive factor \(S^2_{abs}({{\mathbf {b}}})\), which depends on the impact parameter \({{\mathbf {b}}}\) of the \(h_1 h_2\) collision, is included in order to exclude the overlap between the colliding hadrons and insure the dominance of the electromagnetic interaction. Baur and Ferreira-Filho proposed in Ref. [23] to exclude the strong interactions between the incident hadrons by assuming that \(S^2_{abs}({{\mathbf {b}}}) = \Theta \left( \left| {\mathbf{b}}\right| - R_1 - R_2 \right) \) where \(R_i\) are the hadron radii. Such equation treats the incident hadrons as hard spheres with radius \(R_i\) and assumes that the probability to have a hadronic interaction when \(b > R_1 + R_2\) is zero. A more realistic treatment, which takes into account that this probability is finite at \(b > rsim R_1 + R_2\), can be obtained using the Glauber formalism. In this approach the probability of not having a hadronic interaction can be expressed in terms of the nucleon–nucleon cross section and the nuclear overlap function (For details see, e.g. Ref. [24]). As demonstrated in Ref. [25], the impact of the modeling of \(S^2_{abs}\) is negligible for small values of the invariant mass. Following Ref. [26], we will assume that the photon spectrum can be expressed in terms of the electric form factor and that the absorptive corrections \(S^2_{abs}({{\mathbf {b}}})\) for \(\gamma \gamma \) interactions in Pbp and PbPb collisions can be estimated taking into account the multiple scatterings between the nucleons of the incident hadrons, which allows to calculate the probability for no additional hadron–hadron rescattering at different impact parameters. As in Refs. [15, 17], the cross section for the \(\gamma \gamma \rightarrow a \rightarrow \gamma \gamma \) subprocess will be estimated assuming that the ALP is a narrow resonance with a mass \(m_a\) that couples to the \(\gamma \gamma \) system with strength \(g_{a\gamma }\). Similar expressions can be written for the LbL scattering and dilepton production, with the difference that they will be expressed in terms of \(\hat{\sigma }\left( \gamma \gamma \rightarrow \gamma \gamma \right) \) and \(\hat{\sigma }\left( \gamma \gamma \rightarrow e^+ e^- \right) \), respectively (for details see, e.g. Refs. [18, 27]). The signal and the background associated to the LbL scattering will be characterized by a very clean final state, consisting of the diphoton system, two intact hadrons and two rapidity gaps, i.e. empty regions in pseudo-rapidity that separate the intact very forward hadrons from the \(\gamma \gamma \) system. One has that the outgoing hadrons will have very large values of pseudo-rapidity (\(|\eta _i| > rsim 8.0\)), which implies that the rapidity gap between the \(\gamma \gamma \) system and the forward hadrons will be larger than 3.5 units of rapidity, with the minimum occuring for a diphoton produced in the edge of the phase space covered by the LHCb detector. A similar final state will be observed in the dilepton production if the electron and positron are both misidentified as photons. Although the probability of this misidentification is expected to be small (\(\approx 0.5\%\) for each individual lepton), such background can become nonnegligible due to the huge cross section for the dielectron production in UPCs (see e.g. Ref. [27]). As in Refs. [17, 18, 27], we will estimate the total cross sections and differential distributions using the superchicMC event generator [21, 26]. The LbL scattering will be calculated taking into account of the fermion loop contributions as well as the contribution from W bosons, with the later being negligible for the values of \(m_{\gamma \gamma }\) of interest in our analysis. We will focus on the diphoton production with invariant mass in the range \(1 \le m_{\gamma \gamma } \le 20\) GeV, which will allow us to probe the production of ALPs with small mass. In particular, we will present results for eight distinct combinations of mass and coupling, with special emphasis on the mass range \(m_a \le 5\) GeV, which cannot currently be reached by the ATLAS and CMS detectors. We will select events in which \(m_{\gamma \gamma }>\) 1 GeV, \(p_{T}(\gamma ^{1})>\) 0.2 GeV and \(p_{T}(\gamma ^{2})>\) 0.2 GeV, with \(p_T(\gamma ^{i})\) being the transverse momenta of the outgoing photons. In what follows, we will denote the selection in both photons by \(p_{T}(\gamma ^{1},\gamma ^{2})>\) 0.2 GeV. Moreover, we will impose a cut on the acoplanarity (\(1 -(\Delta \phi /\pi ) < 0.01\)) and transverse momentum of the diphoton system [\(p_{T}(\gamma \gamma ) \equiv |\vec {p_T}(\gamma ^{1}) - \vec {p_T}(\gamma ^{2})| < 0.1\) GeV]. As demonstrated in Refs. [17, 18, 27]), such cuts are able to suppress the background associated with gluon-initiated processes and to guarantee the exclusivity criteria. As we are interested in the exclusive ALP production in the kinematical range probed by the LHCb detector, we will select events where photons are produced in the rapidity range \(2.0< |\eta (\gamma ^{1},\gamma ^{2})| < 4.5\) with 0 extra tracks with \(p_{T} > 0.1\) GeV in the range \(5.5< |\eta | < 8.0\), which corresponds to the HERSCHEL acceptance installed in the LHCb experiment [28]. For completeness of our analysis, we also will present predictions for a typical central detector, assuming that it is able to observe events of a diphoton system with a small invariant mass. For this case, we only will select events where photons are produced in the rapidity range \(|\eta (\gamma ^{1},\gamma ^{2})| < 2.5\) with 0 extra tracks.

Table 1 Predictions for the cross sections for the exclusive diphoton production in PbPb collisions at \(\sqrt{s_{_{\textsc {nn}}}}= 5.5\) TeV associated to the ALP production with different masses \(m_a\) and couplings \(g_{a\gamma }\), LbL scattering and exclusive dielectron production adjusted by the misidentification probability for each electron individually
Table 2 Predictions for the cross sections for the exclusive diphoton production in Pbp collisions at \(\sqrt{s_{_{\textsc {nn}}}}= 8.1\) TeV associated to the ALP production with different masses \(m_a\) and couplings \(g_{a\gamma }\), LbL scattering and exclusive dielectron production adjusted by the misidentification probability for each electron individually

In Table 1 we present our predictions for the total cross sections associated to the signal and backgrounds considered in our analysis assuming PbPb collisions at \(\sqrt{s_{_{\textsc {nn}}}}= 5.5\) TeV. The background that comes from photon misidentification due to electrons is denoted by \(e^+ e^- (\gamma \gamma )\), which is estimated assuming that the associated probability is \(0.5\%\) for each individual lepton. We consider four distinct values for the ALP mass \(m_a\) and two possible values for the coupling constant \(g_{a\gamma }\). The chosen \(g_{a\gamma }\) are arbitrary but inside the range not excluded by previous experiments, with the values being mainly motivated by the possibility of complementing the predictions presented in Ref. [17]. The results at the generation level are also presented in order to illustrate the impact of the exclusivity cuts. Our results indicate that the background associated to \(e^+ e^- (\gamma \gamma )\) is strongly suppressed by these cuts. For \(m_a = 2\) GeV, the signal dominates for both values of \(g_{a\gamma }\), with the dominance being larger when an additional cut on the invariant mass is applied. For larger values of the ALP mass, the predictions for the signal and the LbL background are similar, but again the selection of events in a given \(m_{\gamma \gamma }\) range has a large impact, strongly suppressing the LbL background. Similar conclusions are valid for Pbp collisions at \(\sqrt{s_{_{\textsc {nn}}}}= 8.1\) TeV, as observed from the analysis of the results presented in Table 2. The main difference is associated to the fact that the cross sections for Pbp collisions are of the order of pb, while for PbPb they are \({{\mathcal {O}}}\)(nb). Such result is directly associated to the dependence of the photon–photon luminosity on the product \(Z_1^2 Z^2_2\), which implies that the Pbp predictions are suppressed by a factor \(\approx (82)^2\) in comparison to those for PbPb. However, it is important to emphasize that the expected integrated luminosities for the future Pbp runs are, in general, larger than for PbPb collisions, which partially reduces this suppression in the associated number of events and become Pbp collisions competitive to probe the ALP production.

Fig. 2
figure 2

Predictions for the invariant mass \(m_{\gamma \gamma }\), transverse momentum \(p_{T}(\gamma \gamma )\), acoplanarity and rapidity distributions of the diphoton system produced in Pbp collisions at \(\sqrt{s_{_{\textsc {nn}}}}= 8.1\) TeV derived at generation level (left panels) and considering the forward (middle panels) and central (right panels) selections

As the predictions presented in Tables 1 and 2 were calculated using realistic experimental requirements and the associated number of events per year predicted for next run of the LHC are larger than \(10^4\) (\(10^5\)) for \(Pb - p\) (PbPb) collisions, one has that the searching for ALPs with small masses in the exclusive diphoton production is, in principle, feasible by the LHCb collaboration. Motivated by this result, in Fig. 2 (middle panels) we present our predictions for the invariant mass, transverse momentum, acoplanarity and rapidity distributions considering Pbp collisions at \(\sqrt{s_{_{\textsc {nn}}}}= 8.1\) TeV and the forward selection. For completeness, we also present the predictions at the generation level (left panels) and for the central selection (right panels). The results were derived for \(m_a = 2\) GeV and \(g_{a\gamma } = 10^{-3}\) GeV\(^{-1}\). Similar distributions for PbPb collisions can be found in Ref. [17]. One has that the ALP production as a resonance in the s-channel of the \(\gamma \gamma \rightarrow \gamma \gamma \) reaction implies a peak in the invariant mass distribution. In agreement with the results presented in Table 2, the contribution of the \(e^+ e^- (\gamma \gamma )\) background is of the same order of the signal at the generation level, but is strongly suppressed by the exclusivity cuts. Similar suppression is observed for the LbL background. Moreover, the signal dominantes the invariant mass distribution if the range 1.5 GeV \( \le m_{\gamma \gamma } \le 2.5\) GeV is selected, which implies that the study of the diphoton production at small masses is a direct probe of the ALP. The shape of the transverse momentum, acoplanarity and rapidity distributions for the signal and backgrounds are similar, differing in the normalization.

Fig. 3
figure 3

Expected sensitivity to the ALP production in ultraperipheral PbPb (left panel) and Pbp (right panel) collisions at the LHC derived considering different luminosities and selection criteria. Existing exclusion limits derived in \(e^+ e^-\), pp and PbPb collisions are represented by the various shaded regions

Finally, let’s estimate the expected sensitivity to ALPs in the exclusive diphoton production in ultraperipheral Pbp and PbPb collisions considering the current and expected future scenarios for the integrated luminosity. The existing exclusion limits on the ALP parameter space (\(m_a, \, g_{a\gamma }\)) for ALP masses in the range of interest for this study has been derived assuming that \({{\mathcal {B}}}(a \rightarrow \gamma \gamma ) = 1\) and come from \(e^+ e^-\) collisions at the LEP [29] and hadronic collisions at the LHC [12, 13, 30]. They are represented by the shaded regions in Fig. 3. In order to derive our projections, we set a 95\(\%\) confidence limit bound on the signal cross section and remaining backgrounds, assuming Poisson statistics as implemented in previous studies [15, 16]. Our results for PbPb collisions at \(\sqrt{s_{_{\textsc {nn}}}}= 5.5\) TeV and Pbp collisions at \(\sqrt{s_{_{\textsc {nn}}}}= 8.1\) TeV are represented by dashed lines in the left and right panels of the Fig. 3, respectively. For comparison, we also present the projections derived in Refs. [15, 16] for \(m_a \ge 5\) GeV assuming the typical kinematical cuts that can be applied by the ATLAS and CMS detectors. Our results for the forward selection indicate that PbPb collisions provide more stringent constraints on the ALP–photon coupling than Pbp one for masses larger than 2 GeV. However, the analysis of the exclusive diphoton production in Pbp collisions is also competitive to exclude part of the ALP parameter space. These results indicate that a future analysis by the LHCb Collaboration is, in principle, able to enlarge the exclusion limits in a region not covered by previous experiments. For the central selection considered in our analysis, we derive projections slightly distinct from the results obtained in Refs. [15, 16] for larger masses, which is associated to the different cuts implemented in the distinct studies. Our results point out that if a photon pair with low invariant masses could be measured by a central detector, we also shall improve the current bound limits in an unexploited region of the ALP parameter space.

In summary, we have performed a detailed analysis of the ALP production in the kinematical range probed by the LHCb detector and found that the study of the diphoton production in ultraperipheral Pbp and PbPb collisions can improve the limits on the ALP–photon couplings for ALP masses beyond the current reach of central detectors. Our results indicate that the exclusivity cuts suppress the potential backgrounds and that the ALP signal is dominant if the searching is performed for finite ranges of the invariant mass. Predictions for the rapidity, invariant mass, transverse momentum and acoplanarity distributions of the diphoton system produced in Pbp collisions were presented by the first time. Our study demonstrate that a future experimental analysis of the diphoton final state is a promissing observable to probe the existence of the Axionlike particles and its properties.