A Meta-analysis of the 8 TeV ATLAS and CMS SUSY Searches

Between the ATLAS and CMS collaborations at the LHC, hundreds of individual event selections have been measured in the data to look for evidence of supersymmetry at a center of mass energy of 8 TeV. While there is currently no significant evidence for any particular model of supersymmetry, the large number of searches should have produced some large statistical fluctuations. By analyzing the distribution of p-values from the various searches, we determine that the number of excesses is consistent with the Standard Model only hypothesis. However, we do find a shortage of signal regions with far fewer observed events than expected in both the ATLAS and CMS datasets (at $1.65\sigma$ and $2.77\sigma$, respectively). While not as compelling as a surplus of excesses, the lack of deficits could be a hint of new physics already in the 8 TeV datasets.


Introduction
The 2012 discovery of the Higgs boson by the ATLAS [1] and CMS [2] experiments at the Large Hadron Collider (LHC) brought widespread attention to the field of high energy physics. The discovery also reaffirmed the existence of the hierarchy problem due to the small value of the Higgs boson mass. One popular solution to the hierarchy problem is Supersymmetry (SUSY), in which every Standard Model (SM) particle has a supersymmetric counterpart. Since no SUSY particles have been observed, SUSY must be broken: the SUSY partner masses are much larger than their SM analogues. However, the LHC provides a large center of mass energy which could be high enough to produce some of these SUSY particles. Both ATLAS and CMS have conducted extensive searches for SUSY in a multitude of final states, with various numbers of jets, leptons, and photons. The kinematic reach of the detectors have been exploited in order to be sensitive to high mass particles, which may be produced with a low cross section.
Every search for SUSY consists of three pieces: 1) an event selection to maximize sensitivity to a particular model of interest, 2) an estimation of the number of SM events which will pass the selection for a given amount of data, and 3) a comparison of the measured number of events in the region with the predicted number. As the data are stochastic, the last step requires a statistical analysis which incorporates the systematic uncertainties in the predicted event yields. For a particular statistical model of the SM prediction, if the probability p that the measurement could have been generated from the prediction is small, then one has evidence for SUSY. One usually converts the value p into a Gaussian equivalent number of standard deviations and then the generally agreed upon threshold for 'evidence' is 3σ and for 'discovery' is 5σ. However, both ATLAS and CMS have performed many searches. In particular, each analysis usually involves many selections and so there are hundreds of searches between the two collaborations. Statistical fluctuations alone should then lead to several high nσ measurements. By studying the distribution of p-values, we analyze the compatibility of the 8 TeV ATLAS and CMS SUSY searches with the SM-only hypothesis. The procedure is similar to the analysis of the 7 TeV SUSY searches [3], with a few additions that are discussed in the analysis and results sections.

Constructing the Dataset
Even though the 8 TeV dataset was collected in 2012, both ATLAS and CMS are continuing to analyze the data. Most likely, searches will continue to become public until the 14 TeV run efforts are fully operational in the beginning of 2015. Therefore, we arbitrarily cutoff the searches considered for this analysis at the SUSY 2014 conference (July 20, 2014). This includes 17 ATLAS papers [4][5][6][7][8][9][10][11][12][13][14][15][16][17][18][19][20] and 12 CMS papers [21][22][23][24][25][26][27][28][29][30][31][32]. The difficulty in assembling the dataset for the present analysis is to understand the correlations between measurements. The general strategy is to categorize the various searches by their selections on jets, leptons, and photons. Two analyses which have non-overlapping requirements in the number and properties of these objects are treated as uncorrelated. For the data, this is an excellent assumption and only breaks down in the rare case that the data in one signal region is used for the background estimate of another signal region. If two signal regions are such that one is a subset of the other, then a decorrelation procedure is attempted in order to produce two orthogonal regions. If the yields are x ± σ x and y ± σ y with x < y, then the decorrelated regions have yields x ± σ x and (y − x) ± σ 2 y − σ 2 x . In all other cases, it is not possible with the information given to determine the correlations and the signal regions in question are simply not used. In general, if there are two analyses with an unknown correlation, the one with more signal regions is preferred unless the one with fewer regions already has orthogonal selections. Tables 1 and 2 give some summary information about the dataset construction given the general guidelines from above. Multijets Drop all other regions due to unknown correlations.

Analysis
Once the ATLAS and CMS datasets were constructed, the expected and observed distributions of p-values were computed for both a Gaussian and a lognormal distribution of the expected number of counts (the number of counts itself is assumed to be Poisson). A p-value was assigned to each data point according to Here, P ≥n is the probability of observing n or more counts given a Poisson distribution with parameter λ, We performed a similar analysis of deficits rather than excesses in the SUSY search regions by replacing P ≥n in (3.1) with P ≤n : the probability of observing n or less counts given a Poisson distribution with parameter λ. The function φ(λ|µ, σ) is the probability distribution function of the specified random variable with mean µ and standard deviation σ. These parameters are the expected value for the number of counts (µ) and the uncertainty on that value (σ). For the Gaussian distribution, where N is a normalization constant correcting for the fact that λ cannot be negative, and so the negative part of the distribution must be cut off. For the lognormal distribution, whose support is R + , no such normalization constant is required, withμ := ln µ 2 / µ 2 + σ 2 ,σ := ln 1 + σ 2 /µ 2 defined so that the lognormal distribution is precisely the distribution of Y = e X for a Gaussian random variable X with meanμ and varianceσ 2 . One might expect the distribution of p-values defined in this way to be uniformly distributed on the interval [0, 1] under the null hypothesis, in accordance with the usual interpretation of p-values as the probability of observing a more significant result in precisely p × 100% of studies. However, this intuitive understanding is only correct when the distribution is continuous [33], not in the case of Poisson distribution considered here. As a result, we first computed the expected distribution of p-values under the null hypothesis and then compared this with the observed distribution of p-values. The expected distribution of p-values is determined by summing up the probability that each particular trial would fall into one of ten bins, ( i 10 , i+1 10 ], i = 0, ..., 9, Here, X ∼ Poisson(λ) is the random variable measuring the number of counts, and the ≥ in (3.6) is replaced by a ≤ when computing deficits below rather than excesses above the expected signal. Some of the studied signal regions had 0 expected events. There is no lognormal distribution with a mean of 0, so these regions had to be discarded in performing the lognormal analysis. Fortunately, this only applied to seven of the CMS signal regions and none of the ATLAS ones. However, a fairly sizable fraction had an expected mean that was very close to zero. For these trials, it is reasonable to suspect that neither a Gaussian with a cutoff imposed at 0 nor a lognormal will provide a good approximation to the true error distribution. As a double check, we repeated our analysis after removing all data points with µ − 2σ < 0 (≈ 10% for ATLAS, 30% for CMS) . The results of this second analysis did not differ qualitatively from the first, indicating that the results of the original analysis are not significantly affected by the statistical modeling of these data points.

Results and Discussion
The results of our analysis are shown in Figures 1-2. The left-hand side of each plot represents ATLAS data, whereas the right-hand side represents CMS data. The observed distributions for the 2014 ATLAS data look very similar to those for the 2014 CMS data. Both exhibit a lack of deficits with p < 0.1. Statistically, this shortage is only marginally significant for ATLAS with a p-value of 0.10, indicating a lack of deficits at a level of 1.65σ. For CMS, on the other hand, the p-value is 0.006, indicating a shortage of deficits at a level of 2.77σ. Furthermore, the CMS dataset displays a lack of deficits with p < 0.3 at p-value 0.0009, or equivalently a level of 3.32σ. The data also exhibit a statistically significant lack of p-values in the tails of the excess distribution, though this disappears for the ATLAS data once one considers deficits instead of excesses. The greater statistical significance for the CMS distributions compared with the ATLAS distributions may be reflective of the fact that there were many more data points in the CMS dataset compared with the ATLAS dataset.
It is interesting to note that the distributions observed here are somewhat different from those observed in our analysis of the 7 TeV data [3]. That analysis also revealed a deficit of p-values in the tails of the distribution, but there were significantly fewer p-values < 0.1, indicating a possible overestimation of the mean background as well as the uncertainty. Here, there is actually a slight (statistically insignificant) surplus of p-value excesses < 0.1 in the Gaussian case, but a clear lack of p-value deficits < 0.1 in both the Gaussian and lognormal cases. Assuming this is a result of systematic uncertainties rather than unusually large statistical fluctuations, it would indicate one of three things: 1. The uncertainty distributions differ significantly from Gaussian and lognormal distributions with the reported uncertainties and means.
2. The background has been underestimated as a result of biases inherent in the estimation methods.
3. The background has been underestimated as a result of new physics.
The present analysis cannot distinguish between these three possibilities. At the least, the differences indicate that the true uncertainty distributions are not well described by Gaussian or lognormal distributions with the reported means and uncertainties. We therefore encourage future SUSY data searches to publish their uncertainty distributions to ensure proper interpretation of the results and more powerful analyses of the data.

Quantity
Dist 13.65 17.49 0.14 0.04 [5] ATLAS Collaboration, Search for supersymmetry in events with large missing transverse momentum, jets, and at least one tau lepton in 20 fb −1 of √ s=8 TeV proton-proton collision data with the ATLAS detector, 1407.0603.
[6] ATLAS Collaboration, Search for strong production of supersymmetric particles in final states with missing transverse momentum and at least three b-jets at √ s = 8 TeV proton-proton collisions with the ATLAS detector, 1407.0600.
[7] ATLAS Collaboration, Search for top squark pair production in final states with one isolated lepton, jets, and missing transverse momentum in √ s = 8 TeV pp collisions with the ATLAS detector, 1407.0583.
[8] ATLAS Collaboration, Search for the direct production of charginos, neutralinos and staus in final states with at least two hadronically decaying taus and missing transverse momentum in pp collisions at √ s = 8 TeV with the ATLAS detector, 1407.0350. [10] ATLAS Collaboration, Search for squarks and gluinos with the ATLAS detector in final states with jets and missing transverse momentum using √ s = 8 TeV proton-proton collision data, 1405.7875.