An evaluation of percentile measures of citation impact, and a proposal for making them better

Percentiles are statistics pointing to the standing of a paper’s citation impact relative to other papers in a given citation distribution. Percentile Ranks (PRs) often play an important role in evaluating the impact of researchers, institutions, and similar lines of study. Because PRs are so important for the assessment of scholarly impact, and because citations differ greatly across time and fields, various percentile approaches have been proposed to time- and field-normalize citations. Unfortunately, current popular methods often face significant problems in time- and field-normalization, including when papers are assigned to multiple fields or have been published by more than one unit (e.g., researchers or countries). They also face problems for estimating citation counts for pre-defined PRs (e.g., the 90th PR). We offer a series of guidelines and procedures that, we argue, address these problems and others and provide a superior means to make the use of percentile methods more accurate and informative. In particular, we introduce two approaches, CP-IN and CP-EX, that should be preferred in bibliometric studies because they consider the complete citation distribution and can be accurately interpreted. Both approaches are based on cumulative frequencies in percentages (CPs). The paper further shows how bar graphs and beamplots can present PRs in a more meaningful and accurate manner.


Introduction
Citation analyses are frequently used for cross-field and cross-time comparisons.For example, universities are compared in university rankings (e.g., the Leiden Ranking, see https://www.leidenranking.com)which publish in different fields and publication years.This leads to challenges for citation analyses, because of two reasons: first, citation counts depend on the length of the citation window: the longer the time papers can be cited, the more citations can be expected.Second, publication and citation behaviors are different in the fields, why a single citation has a different meaning in each field.Since the 1980s, various methods have been introduced (Tahamtan & Bornmann, 2018) to time-and field-normalize citations.In one of the most frequently used methodsthe relative citation rateexpected values are calculated for every combination of publication year and subject category in databases such as Web of Science (WoS, Clarivate Analytics) or Scopus (Elsevier).These expected values are mean numbers of citations.Then, the citation impact of every focal paper in the combination of publication year and subject category is normalized by dividing the citation counts of the focal paper by the corresponding expected value (Schubert & Braun, 1986).
This approach of generating normalized citation impact values which can be used for cross-time and cross-field comparisons has been frequently criticized, since it is based on arithmetic averages of citations and citations as a rule are skewed distributed.In case of skewed distributions, the arithmetic average should not be used as a measure for the central tendency of the distribution.As an alternative to this normalization approach, various percentile approaches (plotting positions and percentile ranks, PRs) have been proposed.
Plotting positions are quantiles of an empirical (or theoretical) distribution whereby quantiles are defined as specific cut points partitioning distributions into subsets.For example, the median is the 4-quantile separating distributions in four parts.Quantiles can be plotted in so called Q-Q (quantile-quantile) plotsa method for comparing two empirical (or theoretical) distributions.In other words, quantiles of two distributions are plotted against each other in Q-Q plots.
Since a percentile is defined as "a statistic that gives the relative standing of a numerical data point when compared to all other data points in a distribution" (Lavrakas, 2008), plotting positions can be interpreted as percentiles.PR x is defined as the citation count (at or) below which x% (e.g., 90%) of the papers in the combination of publication year and subject category falls.Two papers from different combinations of subject category and publication year with exactly the same PR may have different citation counts.The advantage of PRs (and plotting position) is that they are not affected by outliers (highly cited papers) and their interpretation is simple and clear: if a focal paper has a PR of 90, then 90% of the papers in the publication year and subject category have a citation impact which is (at or) below the impact of the focal paper.This interpretation of PRs makes their use in citation analysis attractive, since it clearly shows the position of the paper in the combination of subject category and publication year which can be compared with the position of other papers.
It is not only possible to calculate plotting positions and PRs for every single paper in a database such as WoS.It is popular in bibliometrics (Bornmann, 2014) to identify in every combination of publication year and subject category the papers which belong to the 10% most frequently cited papers (Bornmann, de Moya Anegón, & Leydesdorff, 2012;Narin, 1987;Waltman et al., 2012).These are the papers in the citation distribution at or above the 90 th PR.The number of these papers can be counted for various units (e.g., for a journal, researcher, university or country).P(top 10%) is the number of top-10% papers and PP(top 10%) is the proportion of top-10% papers published by a unit.Both indicators are presented for every university in the Leiden Ranking.Waltman et al. (2012) regard the PP(top 10%) indicator "as the most important impact indicator in the Leiden Ranking" (p.2425).The Leiden Ranking also provides institutional results based on the 1%, 5%, and 50% most frequently cited papers.
In recent years, various percentile approaches have been introduced in bibliometrics.
In this study, the different approaches are presented, and their advantages and disadvantages explained.Our discussion will show that widely used percentile measures are problematic in some fairly common situations, such as when a paper is classified in more than one field or has been published by more than one unit (e.g., researchers or countries).We then offer an optimized PR approach that maintains most of the advantages of PR measures while overcoming or minimizing their most serious weaknesses.We further show how graphical approaches such as bar graphs and beamplots can make the presentation of PR results more meaningful and accurate.

Methods
For discussing the different percentile approaches, two datasets are used:

Counting highly cited papers
Let us start the discussion of the various percentile approaches with the family of P(top x%) indicators: the number of papers belonging to the x% most-frequently cited papers.
The use of P(top x%) in research evaluation might be interpreted as unsatisfying, since the diverse citation impact of a unit (e.g., a researcher) is transformed into a binary information.
Imagine researcher A has published many papers which are all in the range of P(top 11%) and P(top 31%) and another researcher B who has published all papers in the range of P(top 71%) and P(top 91%).Based on these numbers, one can conclude that researcher A has a better performance than researcher B. However, if only the number of P(top 10%) are counted, both researchers would receive the same assessment with P(top 10%) = 0.
Furthermore, the calculation of P(top x%) is affected by the problem of citation ties at the threshold for separating the x% most frequently cited papers from the rest: suppose five papers with 20 citations, 20 papers with 10 citations, and 75 papers with 1 citations.It is not clear with this citation distribution whether the 20 papers with 10 citations should be assigned to the P(top 10%) or the bottom 90%.Although Waltman and Schreiber (2013) found an elegant solution for that problem (leading to a fractional assignment of papers at the threshold to the group of highly cited papers), it leads to data which are no longer binary: the papers at the threshold are counted with a value less than 1.The consequence is, for instance, that the data can no longer be analyzed with logistic regression analyses, although the nature of the indicator (papers belonging to the top-x% or not) would suggest that this is the appropriate method.
As the example above with researchers A and B reveal, it is desirable to have a percentile solution which is able to reflect the whole range of citation impact received by the papers in a certain publication set.With the integrated impact indicator (I3), Leydesdorff andBornmann (2011, 2012) proposed a solution going beyond the binary classification of impact.
Here, the papers in a set (of a journal or a university) are assigned to more than two impact

4
Plotting positions Bornmann, Leydesdorff, and Mutz (2013) discussed several possibilities to calculate plotting positions for receiving time-and field-normalized citation impact values on the single paper level.They preferred the calculation of plotting position based on the rule proposed by Hazen (1914) using the formula  − 0.5  For calculating the plotting positions, the papers (published in one publication year and subject category) are sorted in decreasing order of citation counts and ranking positions are assigned, whereby i is the rank of the paper and n is the total number of papers in the set.
As the example in Table 1 shows, the formula returns values which are between 0 and 1; for receiving percentages, the values can be multiplied by 100.Ties in citation data do not pose a problem for the calculation, since the corresponding papers simply receive the same (mean) rank and plotting position.Plotting positions can be calculated for the papers in all subject categories and publication years in a database such as WoS, whereby one receives comparable time-and field-normalized citation impact values.For example, the results by Bornmann and Marx (2015) reveal favorable results of plotting positions based on the rule proposed by Hazen (1914) compared to other time-and field-normalization methods (e.g., methods based on mean citations, see above).Hazen (1914) that the formula leads to a value of 0.5 (or 50 as percentage) for the single middle value in the citation distribution.However, this is not always the case as Table 1 demonstrates.Another problem of the plotting positions concerns their interpretation: a usual definition of a PR x is that it represents the citation count at or below which x percent of the papers falls.Four papers in Table 1 have zero citations and a plotting position of 9.52.
Thus, one could assume that around 10% of the papers in the table have zero citations; however, there are around 20% of the papers with zero citations.Plotting positions have been initially proposed and are calculated for the comparison of two empirical distributions (or an empirical distribution with a theoretical distribution), but not for using them for relative assessments.However, the problem with the interpretation of plotting positions especially concerns small publication sets with only a few papers (fewer than 100 papers).Suppose that there are 100 papers in a set with different citation counts each.Then, the plotting positions correspond approximately with the percentage of papers at or below the citation impact of the focal paper.For example, the paper with the 10 th rank position will have the plotting position 0.095.
Figure 1 shows a Q-Q plot of quantiles which have been calculated based on the rule proposed by Hazen (1914).For all papers published between 2000 and 2005 and assigned to six subject categories in WoS (n = 6,070 papers), the quantiles are shown resulting from the first and sixth subject category.Since the general trend of the Q-Q plot is on the line y = x which follows the 45° line, the quantiles in both subject categories 1 and 6 are similar.Thus, most of the papers seem to have the same citation impact relative to other papers in the corresponding subject categories.
Figure 1.Q-Q plot of quantiles based on the rule proposed by Hazen (1914).For all papers published between 2000 and 2005 and assigned to six subject categories, the quantiles are shown resulting from the first and sixth subject category.
Several other rules have been proposed in the past which can be used instead of Hazen (1914) which lead, however, to similar plotting positions (Cox, 2005, discusses some rules).

Approaches based on size-frequency distributions
In recent years, some other approaches have been proposed for citation analyses which might lead to time-and field-normalized citation impact values.Since these approaches are not used for calculating plotting positions, they are explained in this section based on the sizefrequency distribution (Egghe, 2005).This distribution shows the frequencies of papers with certain citation counts in a set of papers (see the first two columns in Table 2).
The column "InCites" in problem with this approach is similar to that of the Hazen approach: a P100 value does not refer to the citation count at or below which x percent of the papers in the combination of publication year and subject category falls.Thus, Bornmann and Mutz (2014) introduced P100' as an alternative to P100 which is calculated using the formula    * 100 whereby j are ranks based on the size-frequency distribution (see Table 2).It is a decisive advantage of P100' that the indicator always has the maximum value 100 and the minimum value 0. However, it remains the problem (as with the P100 and plotting position indicators) that it cannot be interpreted properly: P100' = 21.05does not mean that 21.05% of the papers in the publication set are below 1 citation (or equal to that citation count).

Citation percentiles
The problem of proper interpretations of percentiles can be solved by calculating cumulative frequencies in percentages (CPs) as demonstrated in   PRs such as CP-IN and CP-EX can be calculated for all papers in every combination of subject category and publication year in databases such as WoS.Then, these papers receive time-and field-normalized citation impact values which can be used for various cross-time and cross-field comparisons (e.g., for the comparison of universities or countries).For these and other practical uses of PRs (CP-IN and CP-EX), however, three problems have to be solved.

First problem
The first problem is that the PRs which have been calculated for different combinations of publication years and subject categories are frequently not comparable.As the results in Table 2 demonstrate, we know the citation counts for the PRs 90.48 and 42.86, but we do not know the citation counts for the PRs 90 and 50.In other tables, the citation counts for other PRs might be available (e.g., for 82.34 and 23.45).However, for comparing the impact differences between two subject categories, it is necessary to know these citation counts for predefined PRs (e.g., 90 and 50).Barrett (2003) describes an approach which can be used to solve this problem of comparability.This approach requires that one takes into account the lower and upper bounds for every citation count in a citation distribution assuming that "each exact integer score is actually the middle score of an interval extending 0.5 either side" (p. 6).Furthermore, the observable list of citation counts between the minimum and maximum value in a publication set is complemented by the missing citation counts (with zero numbers of papers).
Table 4 includes CP-IN and shows the same dataset as in the previous tables but considers the lower and upper bounds for each citation count.Furthermore, the missing citation counts (between two observable citation counts) with corresponding zero numbers of papers are added.Thus, a complete list of citation counts beginning with the minimum and ending with the maximum citation counts from the initial publication set with the corresponding numbers of papers is available now.In Table 4, the observed citation counts from the previous tables are expanded to citation-intervals with equal sizes (i.e., 1 citation).The resulting citation counts for pre-specified PRs are "estimates of hypothetical realvalued continuous numbers" (Barrett, 2003, p. 9).These estimates can be calculated for PRs which are between the minimum and maximum CP-IN in Table 4 (i.e., 19.05 and 100).Thus, the citation count for the 4 th PR cannot be estimated.).We can expect that the 90 th PR is in the interval between 12.5 and 13.5 citations; 13 citations refer to the PR 90.48 which is close to 90.The lower bound of the interval is 12.5, i.e. lb = 12.5.We are interested in the 90 th PR, thus p = 0.9, and there are 21 papers in the publication set (n = 21).The width of the class interval is 1 citation (w = 1).The frequency and cumulative frequency of papers containing the PR is fi = 2 and cf = 17.Filling these values in the formula lead to an estimate of 13.45 citations.Since 13 citations correspond to the PR 90.48 and the upper bound of the interval containing this PR is 13.5, 13.45 citations seems to be a realistic value.

Using the frequency distribution in
The calculation of the estimated citation count for the 75 th PR is similar.This PR is between 8 and 9 citations in Table 4. Thus, lb = 8.5.The interval, in which 8.5 is the lower bound, refers to the PR 76.19.The other values for the formula are n = 21, fi = 1, cf = 15, w = 1, and p = 0.75.The estimated result for the 75 th PR is 9.25 which is close to the upper bound of the interval (9.5).This upper bound is from the PR 76.19 which is somewhat higher than 75.The last example is the 50 th PR which is between 7 and 8 citations.Thus, the values for the formula are lb = 6.5, n = 21, fi = 4, cf = 9, w = 1, and p = 0.5.The estimated citation count is 6.875a realistic value with 7.5 as upper bound for the PRs 61.9 and 5.5 as lower bound for the PR 42.86.
Using the formula and the values as specified above, the corresponding PRs (PRx) can be calculated.The only new parameter is x.This is the citation count for which the PR is calculated.Filling in ll The same calculations as with CP-IN can be done with CP-EX but with slightly different formulas.The formula for calculating the citation count for PR i is  5, the citation counts for the 90 th PR is exemplarily calculated.This PR is positioned between 13 citations (CP-EX = 80.95) and 14 citations (CP-EX = 90.48).We select the interval in the table with the exact upper bound value 13.5.The lower bound of the interval is 12.5, i.e. lb = 12.5.The other values are p = 0.9, n = 21, w = 1, fi = 2, and cf = 17.
Filling these values in the formula lead to an estimate of 14.45 citations.Since 14 citations corresponds to the PR 90.48, which is very close to the 90 th PR, and the upper bound of the interval containing this PR is 14.5, 14.45 citations seems to be reasonable.
Using the formula and the values from above, the corresponding PR (PRx) can be calculated.Filling in ll = 6.5, n = 21, fi = 4, cf = 9, w = 1, and x = 7.875 in the formula, we receive the PR 50.

Second problem
Using two formulas proposed by Barrett (2003)   Table 6 shows that 58.49% of the articles are assigned to one subject category.For around 40% of the articles, more than one PR is calculated, and it is not clear how these PRs can be aggregated into one value for a single article.For the InCites tool, the minimum value is used: "the category in which the percentile value is closest to zero is used, i.e. the best performing value" (see https://clarivate.libguides.com/incites_ba/understanding-indicators).
This approach, however, leads to an overestimation of performance if papers are assigned to more than one subject category and the resulting PRs are (very) different.For example, 1,833,224 articles in the dataset of this study are assigned to at least two subject categories.
The minimum of the differences between two PRs of an article is 0, the maximum is 82.9; the mean difference is 7.84 and the median is 5.77.
Another solution for the aggregation into one value could be the median of the PRs.A challenge to this calculation is that subject categories have different numbers of papers.In the dataset of this study, there are 1528 different combinations of subject category and publication year, with a minimum number of papers of 1 and a maximum number of 43,456 (mean = 4564.1).We can assume that the same PR has a higher value in a combination of publication year and subject category with many papers than in a combination with only a few papers.Suppose a paper is assigned to two subject categories with 10 and 3000 papers.To be at the 50 th PR in these subject categories would have very different meanings: in one case, the paper would be better than five other papers and in the other case, it would be better than 1500 other papers.A good estimation of an aggregated PR might be to find the median of PRs where each subject category's PR is reflected several times proportional to the number of papers in that subject category.Thus, the solution might be the median PR which is weighted by the number of papers in the subject categories.This weighted median can be calculated using the formula )) / (68 + 411)] which is also lower than the unweighted PR with 79.5.

Third problem
The third problem which must be solved with the use of PRs is their aggregation if an 'average' PR is desired for a certain publication set (e.g., of a researcher, university or country).Suppose one is interested in an 'average' PR of paper A (CP-EX = 91.86)and paper B (CP-EX = 77.45)from Table 7 and Table 8, since these are two papers from a certain unit.
In these cases, where the aggregation of PRs is desired for a unit, a similar formula can be used as for single papers with different subject categories, whereby single papers with several subject categories are considered multiple times.This weighted median for single units in science can be calculated using the formula extends the calculation by another weight besides the number of subject categories: if papers have been published by more than one unit (e.g., researchers, institutions or countries), the papers should be fractionally assigned to these units.The fractional assignment is included by the notation FR for paper 1 to paper y.

7
Presenting percentile ranks (PRs) The publication set used in this study consists of 4,416,554 articles published between 2000 and 2005.In this section, based on this dataset, some graphs are presented how PRs can be presented for enabling meaningful interpretations of empirical findings.Authors' country information has been added to the articles in the dataset of this study to enable exemplary analyses and comparisons of units.Instead of countries, any other units could have been used (e.g., researchers or institutions).The first proposal for presenting PRs is to show weighted median PRs as bar graphs.
Using the formula in section 6.3, the weighted median PR has been calculated for some countries (by considering three weights: number of papers in a subject category, number of subject categories, and number of countries).The results are shown in Figure 3 the authors recommend to use beamplots instead of single numbers aggregating percentile distributions (such as in Figure 3).It is an advantage of beamplots that not only annual distributions of percentiles can be presented, but also summary statistics (annual and overall medians).However, it should be considered in the use of beamplots that they are especially suitable for small publication sets (i.e., publication sets of single researchers), since beamplots become unreadable for large sets with many publications.As Figure 3 and Figure 4 demonstrate, percentiles can be visualized very differently.
Further possibilities of presenting and statistically analyzing percentiles can be found in Bornmann (2013) and Williams and Bornmann (2014).

Discussion
In a recent study on landmark publications in scientometrics, Tahamtan and Bornmann (2018) worked out that the first method used for time-and field-normalizing citation data was based on percentiles.The introduction of the percentile method to bibliometrics is associated with the name Francis Narin (retired president of CHI Research Inc.).Already at the beginning of the 1980s, McAllister, Narin, and Corrigan (1983) explained citation percentiles as follows: "the pth percentile of a distribution is defined as the number of citations Xp such that the percent of papers receiving Xp or fewer citations is equal to p. Since citation distributions are discrete, the pth percentile is defined only for certain p that occur in the particular distribution of interest" (p.207).Evered, Hamett, and Narin (1989) used the percentage of papers belonging to the 10% most frequently cited papers (named as 'top decile citation performance') to evaluate the citation impact of various institutional units.About 30 years later, Hicks, Wouters, Waltman, de Rijcke, and Rafols (2015) published ten principles to guide research evaluation (using bibliometrics data) in Nature.According to these authors, "normalized indicators are required, and the most robust normalization method is based on percentiles: each paper is weighted on the basis of the percentile to which it belongs in the citation distribution of its field (the top 1%, 10% or 20%, for example)" (p.430).
In this study, various approaches have been presented for using percentiles in research evaluation.A very popular approach today is to present the percentage of papers for a unit subject category and/or has been published by more than one unit (e.g., more than one country).
Section 7 addresses the presentation of PRs: bar graphs and beamplots can be used to present the results for various units.These graph types are only two examples and other options for visualizing results based on PRs exist.Some other options can be found in Bornmann (2013) and Williams and Bornmann (2014).Since the percentile approach has significant advantages against other time-and field-normalizing approaches especially those which are based on mean citation rates, it would be desirable that the percentile approach is more frequently used in bibliometrics studies.Although I3 indicators such as the popular PP(top 10%) are robust indicators compared to other time-and field-normalized indicators, they have the important disadvantage that information is lost from the citation distribution.
Since CP-IN and CP-EX consider the complete citation distribution, they should be preferred in bibliometric studies.
(1) a fictitious small dataset including 21 papers with various citation counts, and (2) all papers published between 2000 and 2005 with the document type "article" and their citation counts until the end of 2018.The publication and citation data are from the Max Planck Society's inhouse database which is based on the WoS.WoS subject categories, which are sets of similar journals, have been used to compute field-normalized citation impact values (plotting positions and PRs).The publication set consists of 6,973,937 articles.However, these are more papers than have been published between 2000 and 2005, since papers which have been assigned to more than one subject category have been considered multiple times (for calculating plotting positions and PRs in every combination of publication year and subject category).Without multiple mentions of papers, the publication set consists of 4,416,554 articles.The articles received between 0 and 67,582 citations until the end of 2018 (median = 14).
Figure 2 (upper left side) shows the distribution of citation counts for the six years using boxplots.It is clearly visible that the distributions are very skewed and characterized by outliers (by a few highly cited articles).

Figure 2 .
Figure 2. Boxplots showing the distributions of citation counts, CP-INs, and CP-EXs (left side) as well as histograms of CP-INs and CP-EXs (right side) for articles published between 2000 and 2005

CCi
= citation count for the i th PR lb = the exact lower bound of the interval containing the PR n = the total number of papers in the publication set p = the proportion corresponding to the desired PR (between 0 and 1, instead of 0 and 100) cf = the cumulative frequency of papers in the interval containing the PR fi = the frequency of papers in the interval containing the i th PR w = the width of the class interval = 6.5, n = 21, fi = 4, cf = 9, w = 1, and x = 6.875, we receive the PR 50.The formula can be used to determine (estimate) the different PRs for the same citation count in different combinations of publication year and subject category.Thus, differences in citation counts between two combinations of publication years and subject categories can be made visible based on exact estimations: what is the PR for the same citation count in two different combinations of publication years and subject categories?

where
CCi = citation count for the i th PR lb = the exact lower bound of the selected interval containing the score x n = the total number of papers in the publication set p = the proportion corresponding to the desired PR (between 0 and 1, instead of 0 and 100) cf = the cumulative frequency of papers in the selected interval fi = the frequency of papers in the selected interval w = the width of the class interval Based on Table as well as two modified formulas, the first problemmissing comparability of PRs resulting from calculations for different combinations of publication year and subject categorycan be solved.Let us go on with the second problem.Many papers in the WoS and Scopus databases (and in other databases) are assigned to more than one subject category.Between 2000 and 2005, the 4,416,554 articles in the dataset of this study are assigned to up to six subject categories (which leads to 6,973,937 articles including multiple occurrences).

Figure 4
Figure 4 shows the beamplot of CP-EXs for articles published between 2000 and 2005

Figure 4 .
Figure 4. PR beamplot (based on CP-EXs) for articles published between 2000 and 2005 by Libyan authors

Table 1 .
Example set of papers for calculating plotting positions (21 papers) Cox (2005)outlined that it is an important advantage of plotting positions based on the rule proposed by

Table 2 .
Table 2 refers to the approach used in the InCites tool which is a citation-based evaluation tool to analyze institutional performance provided by Clarivate Example set of papers for calculating various time-and field-normalized values based on size-frequency distributions (based on the same 21 papers as in Table 1) Analytics (see https://clarivate.com/webofsciencegroup/solutions/incites).InCites percentiles are cumulative percentages of the size-frequency distribution starting with the percentage of papers with the highest citations.For example, 19.05% of the papers in Table2have 13 or more citations; 9.52% of the papers have at least 20 citations.Clarivate Analytics defines InCites percentiles as the "percentage of papers at each level of citation, i.e., the percentage of papers cited more often than the paper of interest" (see https://clarivate.libguides.com/incites_ba/alpha-indicators).Other than the Hazen approach, the InCites approach provides normalized values which can be interpreted as an exact percentage of papers.The problem with the InCites approach is that one does not immediately know, how good the paper is compared to the other papers in the set.Only the subtraction from 100, i.e. (100x), reveals the percentage of papers performing worse than the focal paper.* 100 whereby i is the rank of the citation in the distribution of unique citation values (see Table2).According to Bornmann and Mutz (2014) P100, however, "has undesirable properties which should be avoided … [for example,] the scale value of a paper can increase as a result of the fact that another paper receives an additional citation" (p.1940).Another

Table 3 .
Table3(and by the InCites approach).The table includes two variants: CP-IN is the cumulative percentage of the sizefrequency distribution of papers.For CP-EX, the first possible percentage is set at 0 rather than considering its actual cumulative percentage.Then, the calculation of the cumulative percentage starts with the percentage of the lowest citation count.In this way, CP-EX reveals exactly the percentage of papers with lower citation impact: for example, CP-EX = 90.48means that 90.48% of the papers in the set received a citation impact which is below 20 citations; 19.05% of the papers received less than one citation.CP-IN has a slightly other Cumulative percentages including (CP-IN) or excluding (CP-EX) the number of papers in the row (based on the same 21 papers as in the previous tables) interpretation: 90.48 means that 90.48% of the papers in the set received a citation impact which is at or below 13 citations; 19.05% of the papers received zero citations.CP-IN and CP-EX have been calculated for all articles in the dataset of this study published between 2000 and 2005.

Table 4 .
Expanding the set of papers for calculating "theoretical" citation counts for certain PRs (CP-IN, based on the same 21 papers as in the previous tables) the citation count for a certain PR i can be calculated, where Table 4, the citation counts for the 90 th , 75 th , and 50 th PRs are exemplarily calculated in the following.Let us start with the 90 th PR.Looking at CP-IN in the table, one can see that the 90 th PR is positioned between 12 citations (CP-IN = 80.95) and 13 citations (CP-IN = 90.48 Table 5 shows the same dataset as in Table 4 but it includes CP-EX instead of CP-IN.

Table 6 .
Frequency and percentage of articles with different numbers of subject categories ( 1 *  1 ) + ( 2 *  2 ) + ⋯ + (  *   )  1 +  2 + ⋯ +   whereby PR is the PR, the subscripts SC1 to SCx represent the different subject categories to which a paper is assigned and n is the total number of papers in these subject categories.
Chinaachieved a weighted median PR of 43.54 (CP-IN) and 40.2 (CP-EX) which is below the expected value for an 'average' citation level (50).Thus, China's performance is better than around 40% of the articles published in the same publication year and subject category.
: the USA is the best performing country in this group with a weighted median PR of 58.68 (CP-IN) and 56.65 (CP-EX).Thus, the articles from this country performed (equal to or) better than about 57% (59%) of the articles published in the same publication year and subject category.