Dynamical projections for the visualization of PDFSense data
 422 Downloads
 1 Citations
Abstract
A recent paper on visualizing the sensitivity of hadronic experiments to nucleon structure (Wang et al. in arXiv:1803.02777, 2018) introduces the tool PDFSense which defines measures to allow the user to judge the sensitivity of PDF fits to a given experiment. The sensitivity is characterized by highdimensional data residuals that are visualized in a 3d subspace of the 10 first principal components or using nonlinear embeddings. We show how a tour, a dynamic visualisation of high dimensional data, can extend this tool beyond 3d relationships. This approach enables resolving structure orthogonal to the 2d viewing plane used so far, and hence finer tuned assessment of the sensitivity.
1 Introduction
Many problems in physics can be broadly characterized as a description of a large number of observations with models that contain multiple parameters. It is common practice to perform a global fit to the observations to arrive at the set of parameter values that best fits the data. To understand how well this fit describes the observations, a series of one or twodimensional projections of confidence level regions are usually provided.
It is desirable to visually inspect the results of such fits to gain insight into their structure. One possibility is to directly compare the predictions of different parameter sets in the vicinity of the best fit. A simple algorithm to organise this idea that results in a manageable number of such parameter sets can be constructed using singular value decomposition (SVD). One first decides the confidence level at which to make the desired comparison and quantifies it with the corresponding \(\Delta \chi ^2\) for the appropriate number of parameters being fit, n. The region in parameter space within the desired confidence level is approximately an ndimensional ellipsoid, and SVD provides an ideal set of 2\(\times n\) points on which to evaluate the predictions of the model for visual inspection. These points are given by the intersections of the ellipsoid with its principal axes and clearly provide a minimal sample of parameter space that covers all relevant directions at a desired confidence level.
A tool for the direct visualisation of the high dimensional model predictions thus constructed has existed in the statistics literature for many years, but has not been applied to high energy physics problems recently.^{1} It is called a tour, and is a dynamic visualization of lowdimensional projections of highdimensional spaces. The most recent incarnation of the tool is available in the R [3] package, called tourr [4]. The goal of this paper is to introduce the use of a tour as a visualisation tool for sensitivity studies of parton distribution functions (PDFs) building on the formalism that has been developed over the years by the CTEQ collaboration. It is beyond the scope of this article to provide a detailed analysis of the PDF uncertainties. The choice of this example has two motivations: the PDF fits embody the generic problem of multidimensional fits to large numbers of observables that are common in high energy physics; and Ref. [1] has recently provided the parameter sets for this problem in an initial effort to visualize the PDF fits. Our starting point will be the PDFSense [1] results but our study differs in an important way: PDFSense utilizes the Tensorflow Embedding Projector [5], limiting visualisation to three of the first ten principal components, that is, a 3d subspace, whereas the tour allows us to explore the full space. As we will see here, this allows additional insights into the fits.
Our paper is organised as follows. In Sect. 2 we first describe the problem as formulated in Ref. [1] and we discuss a toy example to illustrate the concepts involved. We then introduce the tour algorithm and its implementation in Sect. 3. Finally we discuss the results obtained by applying tour to the PDFSense dataset in Sect. 4 and present our conclusions in Sect. 5.
2 PDF fits and residuals
The analysis of collider physics results relies on theoretical calculations of crosssections and distributions. Factorization theorems allow us to bypass nonperturbative physics that cannot be calculated from first principles and to describe instead, the initial state of a reaction in terms of parton distribution functions or PDFs. These consist of simple functional forms describing the probability density for finding a given quark or gluon in the proton with a given momentum fraction x, at a given momentum transfer scale Q, in the lowest order approximation. The PDFs used today have been constructed by fitting high energy physics data collected over many years by multiple experiments and are produced by large collaborations. As such, they constitute an ideal example of a multidimensional parameter fit to a large data set to study with a tour.
For our study we will make use of the framework for treating uncertainties of the PDF predictions as has been defined in [6, 7]. The best fit PDF, defined by the set of n parameters \(a^0_i\), is obtained by finding the global minimum of a \(\chi ^2\) function. To study uncertainties in the fit one considers small variations of the parameters around the minimum using a quadratic approximation for the \(\chi ^2\) function written in terms of the Hessian matrix of second derivatives at the minimum, H. The eigenvectors of this matrix provide the principal axes of the confidence level ellipsoids around the global minimum, and one defines a displacement along these directions to find the n dimensional set of points \(a_i\) which provide 2n PDF sets that differ from the best fit by a desired confidence level.
2.1 Simple illustrative example
 1.
point with highest value in \(\delta _1\), found at low x and with small error bar
 2.
point with parametrized highest value in \(\delta _2\), also has the highest value of x
 3.
point that is not well described by the fits, but has small values of \(\delta \)
 4.
point with intermediate value of x and small errors result in larger values in both \(\delta \) directions.
3 Data visualisation
When looking for structure in high dimensional parameter spaces we rely on tools for dimensional reduction and visualisation. Due to the importance of this task, many methods have been developed. Here we give a brief overview of the tools used in the following work. Note that in the following we adopt a broader definition of the word “data” generally used in statistics, which is not restricted to experimental results.
3.1 Dimension reduction
3.1.1 Principal component analysis
In this work we use the standard implementation prcomp in R for the computation of the principal components.
3.1.2 Nonlinear embeddings
It is also common to examine nonlinear mapping of the data points onto a low dimensional embedding. The aim is to preserve multidimensional structure by minimizing the difference in distances in the full parameter space as compared to distances in the low dimensional projection. PCA is a simple member of this more general type of transformation. A widely used method in machine learning is the algorithm called tdistributed stochastic neighbor embedding (tSNE) [11]. It has a goal to cluster similar points together (i.e. points with small Euclidean distance) while separating the individual clusters from one another. This gives appealing and often useful pictures but results should be considered with care as tSNE is a nonlinear transformation and does not preserve original distance. Note that while nonlinear embeddings may be useful in identifying clusters in the data, their interpretation is limited by lack of an analytical description of the transformation. This is not the case for linear transformations such as the PCA, where the transformation can be readily reversed to identify the contribution of the original parameters to a given principal component direction.
3.2 Tour algorithm
3.2.1 Overview
When a data set has more than two parameters, the tour [12] can be used to plot the multiple dimensions. Currently the typical approach is to plot two parameters or pairs of combinations of the parameters. The tour extends this idea to plot all possible combinations. The viewer is provided with a continuous movie of smooth transitions from one combination to another, from which it is possible to extrapolate the shape of the parameter space in highdimensions. Seeing many combinations in quick succession shows the associations between all the parameters.
There are several types of tours. Here we use a grand tour, of projections from ndimensional parameter space to 2d projections space. A projection of data is computed by multiplying an \( m \times n\) data matrix, \(\mathrm{\mathbf{X}}\), having m sample points in n dimensions, by an orthonormal \(n \times d\) projection matrix, \(\mathrm{\mathbf{A}}\), yielding a ddimensional projection. The grand tour is a mechanism for choosing which projections to display, and how the smooth transitions happen. New projections are chosen from all possible projections, and a geodesic interpolation to a target projection provides the smooth transition. The original algorithm is documented in [13]. The implementation used in this paper is from the tourr [4] package in R [3].
The tour shows linear projections of the parameter space. In contrast, methods like tSNE [11] produce nonlinear mappings from high to low dimensional space. The difference is that the shape of the data in highdimensions is preserved by linear projections, but not with nonlinear mappings.
3.2.2 Algorithm
Summary of key findings, comparing observations made with visualising PDFSense results with the TFEP and with additional insights that can be made using tour. A complete list of experimental datasets together with their CTEQ labelling IDs is given in Appendix A
PDFSense & TFEP  Tour  

1  Three clusters can be separated in the visualisation, labelled DIS, VBP and jet cluster. In the selected view the jet cluster is roughly orthogonal to the DIS cluster  We observe the differences in distributions between the three clusters more clearly. Substructure within the clusters is also observed, and studied in some detail 
2  New ATLAS and CMS results will dominate the jet cluster  A more detailed comparison of jet cluster results shows that CMS results are mainly responsible for extending the range, consistent with sensitivity rankings 
3  \(t\bar{t}\) results are characterized by large \(\vec {\delta }\) but there are only a few points and they are found inside the jet cluster  While the \(t\bar{t}\) results follow similar distributions to the jet cluster, they do contain outlying points 
4  Results from semiinclusive charm production at HERA (147) are found to overlap with the DIS and jet clusters  These results do not take significant values in any direction of the \(\vec {\delta }\) space, directional information is misleading here 
5  CCFR/NuTeV dimuon SIDIS results (124–127) are orthogonal, the direction cannot be resolved in the selected view  The tour resolves the orthogonal direction and further allows to identify outlying points 
6  Reciprocated distance as summary statistic to characterize “relevance” of results  We can use the ranking as guidance to select results to highlight in the visualisation to gain understanding of how the summary statistics relate to raw distributions 
 1.
Given a starting \(n\times d\) projection \(\mathrm{\mathbf{A}}_a\), describing the starting plane, create a new target projection \(\mathrm{\mathbf{A}}_z\), describing the target plane. It is important to check that \(\mathrm{\mathbf{A}}_a\) and \(\mathrm{\mathbf{A}}_z\) describe different planes, and generate a new \(\mathrm{\mathbf{A}}_z\) if necessary. To find the optimal rotation of the starting plane into the target plane we need to find the frames in each plane which are the closest.
 2.
Determine the shortest path between frames using singular value decomposition. \(\mathrm{\mathbf{A}}_a'\mathrm{\mathbf{A}}_z=\mathrm{\mathbf{V}}_a\Lambda \mathrm{\mathbf{V}}_z', ~~~\Lambda =\text{ diag }(\lambda _1\ge \dots \ge \lambda _d)\), and the principal directions in each plane are \(\mathrm{\mathbf{B}}_a=\mathrm{\mathbf{A}}_a\mathrm{\mathbf{V}}_a, \mathrm{\mathbf{B}}_z=\mathrm{\mathbf{A}}_z\mathrm{\mathbf{V}}_z\), a withinplane rotation of the descriptive bases \(\mathrm{\mathbf{A}}_a, \mathrm{\mathbf{A}}_z\) respectively. The principal directions are the frames describing the starting and target planes which have the shortest distance between them. The rotation is defined with respect to these principal directions. The singular values, \(\lambda _i, i=1,\dots , d\), define the smallest angles between the principal directions.
 3.
Orthonormalize \(\mathrm{\mathbf{B}}_z\) on \(\mathrm{\mathbf{B}}_a\), giving \(\mathrm{\mathbf{B}}_*\), to create a rotation framework.
 4.
Calculate the principal angles, \(\tau _i = \cos ^{1}\lambda _i, i=1,\dots , d\).
 5.
Rotate the frames by dividing the angles into increments, \(\tau _i(t)\), for \(t\in (0,1]\), and create the ith column of the new frame, \(\mathrm{\mathbf{b}}_i\), from the ith columns of \(\mathrm{\mathbf{B}}_a\) and \(\mathrm{\mathbf{B}}_*\), by \(\mathrm{\mathbf{b}}_i(t) = \cos (\tau _i(t))\mathrm{\mathbf{b}}_{ai} + \sin (\tau _i(t))\mathrm{\mathbf{b}}_{*i}\). When \(t=1\), the frame will be \(\mathrm{\mathbf{B}}_z\).
 6.
Project the data into \(\mathrm{\mathbf{A}}(t)=\mathrm{\mathbf{B}}(t)\mathrm{\mathbf{V}}_a'\).
 7.
Continue the rotation until \(t=1\). Set the current projection to be \(\mathrm{\mathbf{A}}_a\) and go back to step 1.
The data typically needs some standardization or scaling before computing the tour. This is because we are considering linear combinations of the different parameter directions and differences in overall range might otherwise dominate the resulting display.^{3} This can be as simple as centering each variable on 0, and standardizing to a range of − 1 to 1. It could be as severe as sphering the data which in statistics means that the data is transformed into principal components (from elliptical shape to spherical shape). The same term is used for a different type of transformation in other fields, where observations are scaled to fall on a highdimensional sphere, by scaling each observation to have length 1. (An interesting diversion: this type of sphering is the same transformation made on multivariate normal vectors to obtain a point on a sphere, to choose the target planes in the grand tour.)
The initial description of the tour promised display of all possible projections. Theoretically this is true, but practically it would require that the user stay watching forever! However, the coverage of the space is fairly fast, depending on n, and within a short time it is possible to guarantee all possible projections are displayed within an angle of tolerance.
3.2.3 Display
For physics problems, setting \(d=2\) would be most common. The projected data is displayed as a scatterplot of points. It is also possible to overlay confidence regions, or contours. Groups in the data can be highlighted by color. Displaying the combination of variables of a particular projection can be useful to interpret patterns. This can be realized by plotting a circle with segments indicating the magnitude and direction of the contribution, and it is called the axes.
4 Results
4.1 Results from PDFSense & TFEP
For comparison we first reproduce results similar to those found in [1] by using the TFEP software. A selection of four views is shown in Fig. 4, for a complete set of plots related to the PDFSense column in Table 1 we refer the reader to [1]. The selected examples show how the view was chosen based on orthogonality of assigned groups, and how for the example of the jet+\(t\bar{t}\) group the various contributions have been compared.

Relevant information about the distributions is encoded in more than 3 dimensions. This is clear as PCs 3, 5 and 8 have been selected in the visualisation, thus the majority of variation in the data is not captured in Fig. 4. Moreover, the application of tSNE clustering shown in [1] results in a large number of clusters, indicating higher dimensional structure. It would be preferable to display it as a linear projection for which interpretations are straightforward.

The sphering of data points when preparing the PCA visualisation is removing relevant information about the length of the vectors \(\vec {\delta }_i\).

In addition while the online tool allows highlighting of groups it is considerably less flexible in selecting options compared to scripted tools like the tour, limiting the detail in which the results can efficiently be studied.
4.2 Expanded findings made using the tour
We first optimize the number of principal components considered in our study, and then show how the tour results expand on previous observations, as was summarized in Table 1. The mapping from the original \(\delta \) coordinates onto the PCs for all PCAs considered in this work are listed in Appendix B.
4.2.1 PCA, normalisation and variance explained
In the following we study two sets of principal components (PCA1, PCA2), corresponding to the two data preparation choices described above (i.e. PCA1 = centered, and PCA2 = centered and sphered). Results from each are compared. Note that for this problem, the centering has negligible impact on the results as the mean value in each direction \(\delta _{i,l}^{\pm }\) is close to zero.
In the following we want to study a higher dimensional subspace where we base the number of dimensions considered on the results found in Fig. 5.
Cumulative variance as % explained by the first 15 PCs
PCA  1  2  3  4  5  6  7  8  9  10  11  12  13  14  15 

PCA1  12  21  30  37  43  48  53  57  61  65  68  72  75  78  80 
PCA2  12  24  32  40  47  53  59  63  68  71  74  77  80  82  84 
4.2.2 Grand tour result details
A short tour path is generated, of 20 basis planes and associated interpolation between them, of 2 dimensional projections of 6d. This is used to compare between multiple groups. The examples considered are guided by findings in [1] and are summarized in Table 1.
Grouping of data points We first consider a display corresponding to Fig. 4 (left), i.e. the data set is grouped into three main clusters. Selected views from the animation are shown in Fig. 6, PCA1 (left) and PCA2 (right). The same colors used in Ref. [1] indicate the grouping: the DIS cluster is shown in blue, VBP in orange and the jets cluster in red. The first window in the display shows the axes, the other windows show the projected data, where one group is highlighted in color, while the remaining points are shown below in grey for easy comparison. As can be seen from the selected views, in any particular static view it is only possible to separate two of them at a time. The static views are not sufficient to convey the full picture obtained by watching the tour animation which allows to separate all three groups. The tour indicates that there is higher dimensional structure in the data points as can be seen in the linked animation.
The jet cluster In more detail, we investigate the jet cluster. These results are of special interest since they contain indeed the largest data sets to be added in the fit, which were indeed found to be important according to [1]. In addition, the new experimental data from LHC jet measurements is of interest because of possible tensions such as the systematic offsets in opposing directions for different rapidity bins observed in the ATLAS measurements, see [16] for a general discussion of the issue. As pointed out in [16] tensions can be reduced when adapting the treatment of systematic uncertainties, but cannot be fully resolved [17]. As seen above the jet cluster appears to be described in a lower dimensional subspace. Indeed performing PCA on the results in the jet cluster alone we see that the cumulative proportional variance reaches 49/75/91/95 % for PC1/2/3/4 respectively, with the proportional variance dropping to less than 2% for PC5. We therefore study substructure in this 4 dimensional space. While [1] distinguish three types of groups, i.e. “old” jet results (those included in the CT14HERA2 fit), “new” jet results (more recent ATLAS and CMS results) and \(t\bar{t}\), it makes sense to differentiate the LHC results further by experiment and \(\sqrt{s}\) (motivated also by the differences in sensitivities observed in [1]). For simplicity we consider only the results from performing PCA on the centered data shown in Fig. 7 with grouping into: Tevatron (IDs 504, 514), ATLAS7old (535), CMS7old (538), CMS7new (542), ATLAS7new (544), \(t\bar{t}\)energy (565, 567), \(t\bar{t}\)rap (566, 568) and CMS8 (545). Indeed we observe that the Tevatron results as well as the ATLAS results generally fall in the center of the cluster, with exception of some outlying points. On the other hand CMS 7 and 8 TeV results extend in (different) new directions. It is interesting to note that “old” CMS 7 TeV results extend further out than the corresponding “new” ones. In fact while the new measurement extended to higher rapidities and lower values in jet \(p_T\), the old measurement contains higher \(p_T\) bins no longer present in the updated result, which turn out to give large values of \(\vec {\delta }\). Finally for \(t\bar{t}\) results we distinguish the observations binned in energy (\(p_T^t\) or \(m_{t\bar{t}}\)) or rapidity (\(y_{\langle t/\bar{t}\rangle }\) or \(y_{t\bar{t}}\)). We can identify differences between the two groups in the visualisation, however as already noted in [1] the data points are not significantly different from the main jet cluster.

\(y > 2.5\) and \(\mu > 950\) GeV – marked with a star symbol: only one such point is found in the 7 TeV data sets. It occurs in ATLAS7new, it is the last rapidity bin and is clearly outlying (large negative values in PCs 1, 2 and 3). However no particular trend is observed when comparing with points in nearby bins. There are two more such data points in the CMS8 data set, but they do not stand out in \(\delta \) space.

\(y > 2\) and \(\mu > 1000\) GeV – marked with downward pointing triangle. These points are seen to align in a new direction, away from the main cluster highlighting their importance in the fits. They are also useful for comparing the different CMS results: in this case there are common points to both datasets that nevertheless look different, suggesting the need for further study of these points.

for CMS8 we also highlight \(y < 1\) and \(\mu < 200\) – marked with diamond symbol: they are very different from the main distribution and give large positive values in PC1. It is interesting that we can clearly separate these low \(\mu \) bins in CMS8 set but not in CMS7.
We therefore compare in detail these three groups. In this case it is useful to consider both PCA1 and PCA2, the latter more closely related to the TFEP output. First, we observe that the dimuon SIDIS is poorly separated in the PCA2 projection, whereas PCA1 clearly shows how it extends considerably away from the main DIS cluster (ID 160). On the other hand, the charm SIDIS can be separated more easily when studying the directional information in the PCA2 projection because the individual values in the space of deltas are all comparatively small. These results suggest that either predictions for these type of observables are well under control in the existing fits, or that alternatively the experimental errors are too large for them to be constraining. We also observe substructure in the DIS HERA1+2, see Fig. 8 and the corresponding animation, indicating that this group combines a number of qualitatively different types of results.
Indeed we find that the W asymmetry measurements (234, 266 and 281) follow a very distinct distribution, as does the HERA DIS dataset (160). On the other hand, the fixedtarget DrellYan measurements (201, 203 and 204), do not stand out in our visualisation. We find that this is a consequence of the dimension reduction,^{4} and we can easily identify views separating this group from the other data points when considering additional dimensions. Here we show this by looking at projections found by performing PCA on this data subset only and using it to compare it to the other data sets in the subspace of the first 4 PCs thus defined. Note however that the tour allows visualisation of the distributions in the full parameter space which would yield the same information. Our choice of procedure is simply to limit the viewing times required, which grow with the number of dimensions considered.^{5}
This type of visualisation, together with inverting the mapping onto principal components, may be used to identify the origin (i.e. underlying physics) of the large differences. For example the first three PCs found for the DY dataset capture three different distributions, and mapping those back to the original \(\delta \) directions together with study of those directions with respect to uncertainty in individual parton pdfs may provide additional insight. Such detailed investigations are however beyond the scope of this study.
5 Summary and conclusions
Starting from the set of 56 dimensional vectors in the space of residual responses calculated in [1], we have demonstrated how the grand tour may be used for visualizations in particle physics. The 56 dimensions are reduced to 6 dimensions (for illustration) using principal component analysis, and the resulting representation is then passed onto the tour. The findings made about the fits using the tour, even with only 6 dimensions, are more comprehensive and clearer than what TFEP allows.
The tour visualisation verified several results from [1], notably, the separation between DIS, VBP and JET experiments into clusters populating different regions of delta space. It also allowed us to go into further detail by examining certain substructures within these groups. We have moreover demonstrated that the tour can complement and support analyses based on the use of reciprocated distances.
In our examples we have considered performing the PCA either on centered data (PCA1) or on centered and sphered data (PCA2), as they highlight different aspects of the structure, the former retaining length information and the latter emphasizing directionality. In general we find the results from PCA1 more useful, in particular for this application where the length of the individual data point vectors (i.e. for each experiment) carries important information that is lost when sphering the input data.
Experimental datasets considered as part of CT14HERA2 and included in the analysis. IDs are following the standard CTEQ labelling system with 1XX/2XX/5XX representing datasets in the DIS/VBP/JET group
ID#  Experimental dataset  Group  

101  BCDMS \(F_{2}^{p}\)  [18]  DIS 
102  BCDMS \(F_{2}^{d}\)  [19]  DIS 
104  NMC \(F_{2}^{d}/F_{2}^{p}\)  [20]  DIS 
108  CDHSW \(F_{2}^{p}\)  [21]  DIS 
109  CDHSW \(F_{3}^{p}\)  [21]  DIS 
110  CCFR \(F_{2}^{p}\)  [22]  DIS 
111  CCFR \(xF_{3}^{p}\)  [23]  DIS 
124  NuTeV \(\nu \mu \mu \) SIDIS  [24]  DIS 
125  NuTeV \(\bar{\nu }\mu \mu \) SIDIS  [24]  DIS 
126  CCFR \(\nu \mu \mu \) SIDIS  [25]  DIS 
127  CCFR \(\bar{\nu }\mu \mu \) SIDIS  [25]  DIS 
145  H1 \(\sigma _{r}^{b}\) (\(57.4 \text{ pb }^{1}\))  DIS  
147  Combined HERA charm production (\(1.504 \text{ fb }^{1}\))  [28]  DIS 
160  HERA1+2 Combined NC and CC DIS (\(1 \text{ fb }^{1}\))  [29]  DIS 
169  H1 \(F_{L}\) (\(121.6 \text{ pb }^{1}\))  [30]  DIS 
201  E605 DY  [31]  VBP 
203  E866 DY, \(\sigma _{pd}/(2\sigma _{pp})\)  [32]  VBP 
204  E866 DY, \(Q^{3}d^{2}\sigma _{pp}/(dQdx_{F})\)  [33]  VBP 
225  CDF Run1 \(A_{e}(\eta ^{e})\) (\(110 \text{ pb }^{1}\))  [34]  VBP 
227  CDF Run2 \(A_{e}(\eta ^{e})\) (\(170 \text{ pb }^{1}\))  [35]  VBP 
234  D\(\emptyset \) Run2 \(A_{\mu }(\eta ^{\mu })\) (\(0.3 \text{ fb }^{1}\))  [36]  VBP 
240  LHCb 7 TeV W / Z muon forward\(\eta \) Xsec (\(35 \text{ pb }^{1}\))  [37]  VBP 
241  LHCb 7 TeV W \(A_{\mu }(\eta ^{\mu })\) (\(35 \text{ pb }^{1}\))  [37]  VBP 
260  D\(\emptyset \) Run2 Z \(d\sigma /dy_{Z}\) (\(0.4 \text{ fb }^{1}\))  [38]  VBP 
261  CDF Run2 Z \(d\sigma /dy_{Z}\) (\(2.1 \text{ fb }^{1}\))  [39]  VBP 
266  CMS 7 TeV \(A_{\mu }(\eta )\) (\(4.7 \text{ fb }^{1}\))  [40]  VBP 
267  CMS 7 TeV \(A_{e}(\eta )\) (\(0.840 \text{ fb }^{1}\))  [41]  VBP 
268  ATLAS 7 TeV W / Z Xsec, \(A_{\mu }(\eta )\) (\(35 \text{ pb }^{1}\))  [42]  VBP 
281  D\(\emptyset \) Run2 \(A_{e}(\eta )\) (\(9.7 \text{ fb }^{1}\))  [43]  VBP 
504  CDF Run2 incl. jet (\(d^2\sigma /dp_{T}^{j}dy_{j}\)) (\(1.13 \text{ fb }^{1}\))  [44]  JET 
514  D\(\emptyset \) Run2 incl. jet (\(d^2\sigma /dp_{T}^{j}dy_{j}\)) (\(0.7 \text{ fb }^{1}\))  [45]  JET 
535  ATLAS 7 TeV incl. jet (\(d^2\sigma /dp_{T}^{j}dy_{j}\)) (\(35 \text{ pb }^{1}\))  [46]  JET 
538  CMS 7 TeV incl. jet (\(d^2\sigma /dp_{T}^{j}dy_{j}\)) (\(5 \text{ fb }^{1}\))  [47]  JET 
We conclude that the above described method is a valuable tool for PDF uncertainty and sensitivity studies. In addition, the visual analysis allows a better understanding of the method itself and can uncover unexpected features, and even possibly errors. It can provide experiments with a guide to the measurements needed to improve PDF fits.
Footnotes
 1.
The precursor [2] of this tool was originally developed to tackle problems in high energy physics.
 2.
Note that the shifted central data value enters the residuals, thus while the observed central value cancels in the definition of \(\delta _{i,l}^{\pm }\), differences in the shift arising from differences of the optimized nuisance parameters at \(\vec {a}_l^{\pm }\) are encoded in the results together with difference in theory predictions.
 3.
Such a standardisation is in fact done routinely by selecting an axis scale appropriate to the relevant range in each parameter direction in a 2d plot.
 4.
Recall that the selected first six PCs only capture 48% of overall variance.
 5.
When working in the full parameter space one should consider the definition of projection pursuit indices to guide the tour to interesting views, one may e.g. define an index that finds views where a selected group of data points is maximally separated from the cluster of points, similar to the definition of reciprocated distances.
Notes
Acknowledgements
This work was supported in part by the Australian Research Council. We thank Nicholas Spyrison for help with the animations and Timothy Hobbs and Fred Olness for clarifications on their work.
References
 1.B.T. Wang, T. J. Hobbs, S. Doyle, J. Gao, T.J. Hou, P. M. Nadolsky et al., Visualizing the sensitivity of hadronic experiments to nucleon structure. (2018). arXiv:1803.02777
 2.M. Fisherkeller, J. H. Friedman, J. Tukey, PRIM9: An Interactive Multidimensional Data Display and Analysis System. ASA Statistical Graphics Video Lending Library. http://statgraphics.org/movies/prim9.html) (1973)
 3.R Core Team, R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria, 2018Google Scholar
 4.H. Wickham, D. Cook, H. Hofmann, A. Buja, tourr: an R package for exploring multivariate data with projections. J. Stat. Softw. 40, 1 (2011)Google Scholar
 5.TensorFlow Embedding Projector. http://projector.tensorflow.org
 6.J. Pumplin, D.R. Stump, W.K. Tung, Multivariate fitting and the error matrix in global analysis of data. Phys. Rev. D 65, 014011 (2001). arXiv:hepph/0008191 ADSCrossRefGoogle Scholar
 7.J. Pumplin, D. Stump, R. Brock, D. Casey, J. Huston, J. Kalk, Uncertainties of predictions from parton distribution functions. 2. The Hessian method. Phys. Rev. D 65, 014013 (2001). arXiv:hepph/0101032 ADSCrossRefGoogle Scholar
 8.D. Stump, J. Pumplin, R. Brock, D. Casey, J. Huston, J. Kalk, Uncertainties of predictions from parton distribution functions. 1. The Lagrange multiplier method. Phys. Rev. D 65, 014012 (2001). arXiv:hepph/0101051 ADSCrossRefGoogle Scholar
 9.V.D. Barger, W.Y. Keung, R.J.N. Phillips, On psi and upsilon production via gluons. Phys. Lett. 91B, 253 (1980)ADSCrossRefGoogle Scholar
 10.T.J. Hou et al., Reconstruction of Monte Carlo replicas from Hessian parton distributions, JHEP 03 (2017) 099. arXiv:1607.06066
 11.L. van der Maaten, G. Hinton, Visualizing data using tSNE. J. Mach. Learn. Res. 9, 2579 (2008)zbMATHGoogle Scholar
 12.D. Asimov, The grand tour: a tool for viewing multidimensional data. SIAM J. Sci. Stat. Comput. 6, 128 (1985)MathSciNetCrossRefGoogle Scholar
 13.A. Buja, D. Cook, D. Asimov and C. Hurley, 14—Computational Methods for HighDimensional Rotations in Data Visualization, vol. 24 of Handbook of Statistics, pp. 391–413. Elsevier (2005). https://doi.org/10.1016/S01697161(04)240147 Google Scholar
 14.D. Cook, E.K. Lee, A. Buja and H. Wickham, Grand Tours, Projection Pursuit Guided Tours and Manual Controls, ch. III.2, pp. 295–314. Springer Handbooks of Computational Statistics. Springer (2008)Google Scholar
 15.T.J. Hou, S. Dulat, J. Gao, M. Guzzi, J. Huston, P. Nadolsky et al., CTEQTEA parton distribution functions and HERA Run I and II combined data, Phys. Rev. D 95, 034003 (2017). arXiv:1609.07968
 16.L. A. HarlandLang, A. D. Martin, R. S. Thorne, The impact of LHC Jet Data on the MMHT PDF Fit at NNLO, Eur. Phys. J. C 78, 248 (2018). arXiv:1711.05757
 17.ATLAS collaboration, M. Aaboud et al., Measurement of the inclusive jet crosssections in protonproton collisions at \( \sqrt{s}=8 \) TeV with the ATLAS detector, JHEP 09, 020 (2017). arXiv:1706.03192
 18.BCDMS collaboration, A. C. Benvenuti et al., A high statistics measurement of the proton structure functions F(2) (x, Q**2) and R from deep inelastic Muon scattering at high Q**2, Phys. Lett. B 223, 485 (1989)Google Scholar
 19.BCDMS collaboration, A. C. Benvenuti et al., A high statistics measurement of the deuteron structure functions F2 (X, \(Q^2\)) and R from deep inelastic Muon scattering at high \(Q^2\), Phys. Lett. B 237, 592 (1990)Google Scholar
 20.Nucl. Phys. M. Arneodo et al., Measurement of the proton and deuteron structure functions, F2(p) and F2(d), and of the ratio sigmaL / sigmaT. B 483, 3 (1997). arXiv:hepph/9610231
 21.J.P. Berge et al., A measurement of differential crosssections and nucleon structure functions in charged current neutrino interactions on iron. Z. Phys. C 49, 187 (1991)CrossRefGoogle Scholar
 22.CCFR/NuTeV collaboration, U.K. Yang et al., Measurements of \(F_2\) and \(xF^{\nu }_3  x F^{\bar{\nu }}_3\) from CCFR \(\nu_\mu \)Fe and \(\bar{\nu }_\mu \)Fe data in a physics model independent way, Phys. Rev. Lett. 86, 2742 (2001). arXiv:hepex/0009041
 23.W.G. Seligman et al., Improved determination of alpha(s) from neutrino nucleon scattering. Phys. Rev. Lett. 79, 1213 (1997). arXiv:hepex/9701017 ADSCrossRefGoogle Scholar
 24.D. A. Mason, Measurement of the strange—antistrange asymmetry at NLO in QCD from NuTeV dimuon data, Ph.D. thesis, Oregon U., 10.2172/879078 (2006)Google Scholar
 25.NuTeV collaboration, M. Goncharov et al., Precise measurement of Dimuon production crosssections in \(\nu_{\mu }\) Fe and \(\bar{\nu }_{\mu }\) Fe deep inelastic scattering at the tevatron., Phys. Rev. D 64, 112006 (2001). arXiv:hepex/0102049
 26.H1 collaboration, A. Aktas et al., Measurement of F2(\(c \bar{c}\)) and F2(\(b \bar{b}\)) at high \(Q^{2}\) using the H1 vertex detector at HERA, Eur. Phys. J. C 40, 349 (2005). arXiv:hepex/0411046
 27.H1 collaboration, A. Aktas et al., Measurement of F(2)**c antic and F(2)**b antib at low Q*2 and x using the H1 vertex detector at HERA, Eur. Phys. J. C 45, 23 (2006). arXiv:hepex/0507081
 28.ZEUS, H1 collaboration, H. Abramowicz et al., Combination and QCD analysis of charm production cross section measurements in deepinelastic ep scattering at HERA, Eur. Phys. J. C 73, 2311 (2013). arXiv:1211.1182
 29.ZEUS, H1 collaboration, H. Abramowicz et al., Combination of measurements of inclusive deep inelastic \({e^{\pm }p}\) scattering cross sections and QCD analysis of HERA data, Eur. Phys. J. C 75, 580 (2015). arXiv:1506.06042
 30.H1 collaboration, F. D. Aaron et al., Measurement of the inclusive \(e{\pm }p\) scattering cross section at high inelasticity y and of the structure function \(F_L\), Eur. Phys. J. C 71, 1579 (2011). arXiv:1012.4355
 31.G. Moreno et al., Dimuon production in proton–copper collisions at \(\sqrt{s}\) = 38.8GeV. Phys. Rev. D 43, 2815 (1991)ADSCrossRefGoogle Scholar
 32.collaboration, R. S. Towell et al., Improved measurement of the antid / antiu asymmetry in the nucleon sea, Phys. Rev. D 64, 052002 (2001). arXiv:hepex/0103030
 33.NuSea collaboration, J. C. Webb et al., Absolute DrellYan dimuon crosssections in 800 GeV / c pp and pd collisions. arXiv:hepex/0302019
 34.CDF collaboration, F. Abe et al., Forwardbackward charge asymmetry of electron pairs above the \(Z^0\) pole, Phys. Rev. Lett. 77, 2616 (1996)Google Scholar
 35.CDF collaboration, D. Acosta et al., Measurement of the forwardbackward charge asymmetry from \(W \rightarrow e \nu \) production in \(p\bar{p}\) collisions at \(\sqrt{s} = 1.96\) TeV, Phys. Rev. D 71, 051104 (2005) . arXiv:hepex/0501023
 36.D0 collaboration, V. M. Abazov et al., Measurement of the muon charge asymmetry from \(W\) boson decays. Phys. Rev. D 77, 011106 (2008). arXiv:0709.4254
 37.LHCb collaboration, R. Aaij et al., Inclusive \(W\) and \(Z\) production in the forward region at \(\sqrt{s} = 7\) TeV, JHEP 06, 058 (2012). arXiv:1204.1620
 38.D0 collaboration, V. M. Abazov et al., Measurement of the ratios of the Z/gamma* + >= n jet production cross sections to the total inclusive Z/gamma* cross section in p antip collisions at s**(1/2) = 1.96TeV. Phys. Lett. B 658, 112 (2008). arXiv:hepex/0608052
 39.CDF collaboration, T. A. Aaltonen et al., Measurement of \(d\sigma /dy\) of DrellYan \(e^+e^\) pairs in the \(Z\) mass region from \(p\bar{p}\) Collisions at \(\sqrt{s}=1.96\) TeV, Phys. Lett. B 692, 232 (2010). arXiv:0908.3914
 40.CMS collaboration, S. Chatrchyan et al., Measurement of the muon charge asymmetry in inclusive \(pp \rightarrow W+X\) production at \(\sqrt{s} =\) 7 TeV and an improved determination of light parton distribution functions, Phys. Rev. D 90, 032004 (2014). arXiv:1312.6283
 41.CMS collaboration, S. Chatrchyan et al., Measurement of the electron charge asymmetry in inclusive \(W\) production in \(pp\) collisions at \(\sqrt{s}=7\) TeV. Phys. Rev. Lett. 109, 111806 (2012). arXiv:1206.2598
 42.ATLAS collaboration, G. Aad et al., Measurement of the inclusive \(W^\pm \) and Z/gamma cross sections in the electron and muon decay channels in \(pp\) collisions at \(\sqrt{s}=7\) TeV with the ATLAS detector, Phys. Rev. D 85, 072004 (2012). arXiv:1109.5141
 43.D0 collaboration, V. M. Abazov et al., Measurement of the electron charge asymmetry in \(\varvec {p\bar{p}\rightarrow W+X \rightarrow e\nu +X}\) decays in \(\varvec {p\bar{p}}\) collisions at \(\varvec {\sqrt{s}=1.96}\) TeV. Phys. Rev. D 91, 032007 (2015). arXiv:1412.2862
 44.CDF collaboration, T. Aaltonen et al., Measurement of the inclusive jet cross section at the fermilab tevatron p antip collider using a conebased jet algorithm, Phys. Rev. D 78, 052006 (2008). arXiv:0807.2204
 45.D0 collaboration, V. M. Abazov et al., Measurement of the inclusive jet crosssection in \(p \bar{p}\) collisions at \(s^{(1/2)}\) =1.96TeV. Phys. Rev. Lett. 101, 062001 (2008). arXiv:0802.2400
 46.ATLAS collaboration, G. Aad et al., Measurement of inclusive jet and dijet production in \(pp\) collisions at \(\sqrt{s}=7\) TeV using the ATLAS detector, Phys. Rev. D 86, 014022 (2012). arXiv:1112.6297
 47.CMS collaboration, S. Chatrchyan et al., Measurements of differential jet cross sections in protonproton collisions at \(\sqrt{s}=7\) TeV with the CMS detector, Phys. Rev. D 87, 112002 (2013). arXiv:1212.6660
 48.LHCb collaboration, R. Aaij et al., Measurement of the forward \(Z\) boson production crosssection in \(pp\) collisions at \(\sqrt{s}=7\) TeV, JHEP 08, 039 (2015). arXiv: 1505.07024
 49.LHCb collaboration, R. Aaij et al., Measurement of forward \(\rm Z\rightarrow e^+e^\) production at \(\sqrt{s}=8\) TeV, JHEP 05, 109 (2015). arXiv:1503.00963
 50.ATLAS collaboration, G. Aad et al., Measurement of the \(Z/\gamma ^*\) boson transverse momentum distribution in \(pp\) collisions at \(\sqrt{s}\) = 7 TeV with the ATLAS detector, JHEP 09, 145 (2014). arXiv:1406.3660
 51.CMS collaboration, V. Khachatryan et al., Measurement of the differential cross section and charge asymmetry for inclusive \({{\rm pp}\rightarrow {\rm W}^{\pm }+X}\) production at \({\sqrt{s}} = 8\) TeV, Eur. Phys. J. C 76, 469 (2016). arXiv:1603.01803
 52.LHCb collaboration, R. Aaij et al., Measurement of forward W and Z boson production in \(pp\) collisions at \( \sqrt{s}=8 \) TeV, JHEP 01, 155 (2016). arXiv:1511.08039
 53.ATLAS collaboration, G. Aad et al., Measurement of the doubledifferential highmass DrellYan cross section in pp collisions at \( \sqrt{s}=8 \) TeV with the ATLAS detector, JHEP 08, 009 (2016). arXiv:1606.01736
 54.ATLAS collaboration, G. Aad et al., Measurement of the transverse momentum and \(\phi ^*_{\eta }\) distributions of DrellYan lepton pairs in protonproton collisions at \(\sqrt{s}=8\) TeV with the ATLAS detector, Eur. Phys. J. C 76, 291 (2016). arXiv:1512.02192
 55.CMS collaboration, S. Chatrchyan et al., Measurement of the ratio of inclusive jet cross sections using the anti\(k_T\) algorithm with radius parameters R=0.5 and 0.7 in pp collisions at \(\sqrt{s}=7\) TeV, Phys. Rev. D 90, 072006 (2014). arXiv:1406.0324
 56.ATLAS collaboration, G. Aad et al., Measurement of the inclusive jet crosssection in protonproton collisions at \(\sqrt{s}=7\) TeV using 4.5 fb1 of data with the ATLAS detector, JHEP 02, 153 (2015). arXiv:1410.8857
 57.CMS collaboration, V. Khachatryan et al., Measurement and QCD analysis of doubledifferential inclusive jet cross sections in pp collisions at \( \sqrt{s}=8 \) TeV and cross section ratios to 2.76 and 7 TeV, JHEP 03, 156 (2017). arXiv:1609.05331
 58.ATLAS collaboration, G. Aad et al., Measurements of topquark pair differential crosssections in the lepton+jets channel in \(pp\) collisions at \(\sqrt{s}=8\) TeV using the ATLAS detector, Eur. Phys. J. C 76, 538 (2016). arXiv:1511.04716
Copyright information
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.
Funded by SCOAP^{3}