Introduction

Scientific output can be summarized by many ways and via different approaches, including the number of papers, the number of papers in which a given author is either the first or a senior author, the total citation index value, the number of highly cited papers, e.g. with more than 100 citations, the citation index after exclusion of self-citations or the citation index after exclusion of the citations of all co-authors. All these criteria address mainly the quantitative aspects of the scientific output of an individual researcher, while not targeting its qualitative aspects. The first attempt to evaluate the achieved quantity and quality in the scientific activity of researchers was proposed by Hirsch (2005). The now widely used h-index is an original, simple indicator, characterizing the cumulative impact of the research work of individual scientists.

More recently, we have developed a scientific quality index (SQI), a novel indicator of the scientific output of an individual researcher (Pluskiewicz and Drozdzowska 2017). The Scientific Quality Index (SQI) has been defined, according to the following formula: [the percent of papers cited ≥ 10 times versus all the published papers, including those with no citation] + [the mean number of citations per paper, regarding all the published papers, including those with no citation]. All the self-citations and the citations of all co-authors are excluded. The concept of the index has emerged from a perceived need to employ a quality related process in the appraisal of individual scientific achievements.

In order to prove the hypothesis that the h-index and the SQI address different features in scientific outputs, the two bibliometric parameters were compared for a group of 40 globally top-ranked scientists in the field of osteoporosis and the results of that comparison were published in 2018 (Pluskiewicz et al. 2018). In that study, the qualitative features of scientific output, as reflected by the SQI, changed the classification of 37, out of 40 (92.5%) authors, in comparison to the classification provided by the h-index. Those data clearly indicate that the SQI classifies the author’s ranking position differently versus the h-index.

The goal of this study was to measure the scientific output of 480 authors—leaders in 12 selected branches of medicine, using both the h-index and the SQI.

Methods

Data for the 480 researchers, representing 12 selected areas of medical sciences, were derived from the Scopus database. The period of 10 years (2008–2017) was taken into account. Each scientific field was represented by 40 authors with the highest number of published papers, recorded during those 10 years.

The following subgroups of researchers (top 40 authors in each category) were evaluated, according to the key words: allergen, brain, endocrine/hormone, gene, heart, kidney, liver, lung, neoplasm, pregnancy, stomach, surgery. The data were collected in April 2018.

The following bibliometric parameters were applied: the h-index for all citations, the h-index, calculated after the exclusion of self-citations and of the citations of all co-authors, the citation index (the number of citations), except of the citations by the first author and by all of his/her co-authors, the number of all published papers, the number of cited papers, the number of papers cited at least 10 times and the percent of papers, cited at least 10 times among all the published papers, including those with no citation.

Using the bibliometric parameters and data, derived from the Scopus database, the new index—SQI was calculated for each of the analyzed scientists, according to the following formula: Parameter No. 1 + Parameter No. 2, where:

  • Parameter No. 1 (the percent of papers cited ≥ 10 times) = the number of papers cited ≥ 10 times (excluding self-citations and citations of all co-authors) divided by the number of all the published papers (including the papers with no citation) × 100%,

  • Parameter No. 2 (the mean number of citations per paper) = the total number of citations (excluding self-citations and citations of all co-authors) divided by the number of all published papers (including papers with no citation).

As the SQI calculation assumes the exclusion of self-citations and of the citations of all co-authors, we also made use of the h-index, calculated without both self-citations and the citations of all co-authors.

Statistics

The Statistica software (StatSoft, Tulsa, OK, USA) was applied for statistical analysis. Descriptive statistics were presented as mean values and standard deviations. The normality of distribution of the analyzed data was verified, using the Shapiro–Wilk test. A correlation analysis was performed by means of the Spearman’s rank correlation test and the coefficients of correlation were compared by the Fisher exact test. p values below 0.05 were considered statistically significant.

Results

The mean values of the basic bibliometric parameters, describing the individual scientific outputs of selected 480 scientists, were as follows: the citation index: 7250 ± 7817, the total number of papers: 187 ± 104, the total number of cited papers: 175 ± 101, the number of papers cited at least 10 times: 110 ± 75, the percent of papers cited at least 10 times: 51 ± 16, the mean number of citations per paper: 28 ± 21. The mean value of the Hirsch index was 33.2 ± 16 and the mean SQI was 78.5 ± 33.4.

When the evaluated investigators were ranked by decreasing SQI values, only two authors maintained their baseline positions in ranking by decreasing h-index values, whereas 279 (58.1%) authors were ranked lower and 199 (41.5%) higher. The highest improved ranking position resulted from a shift upwards by 387 positions, and the biggest drop demonstrated a fall down by 249 positions.

The mean h-index and SQI values, obtained for the cohorts of forty investigators, each cohort representing a particular branch of medicine, are presented in Fig. 1. The first two positions (in the h-index and SQI ranking) remained unchanged for ‘gene’ and ‘heart’ as the keywords.

Fig. 1
figure 1

The mean values (with error bars presenting SD values) of the h-index and SQI in categories according to selected key words. The order of medical branches on the horizontal axis shows their ranking position by the h-index. The numbers in parentheses correspond to the ranking position of the scientific area by the SQI

Full size image

Table 1 presents individual SQI and h-index values for ten top-ranked researchers, screened by the SQI ranking, including six geneticians, one hepatologist, one endocrinologist, one nephrologist and one cardiologist. The highest SQI value of 234.8 was achieved by Prof. Yusuf Salim, a cardiologist from Canada.

Table 1 Ten top-ranked researchers from the whole study group, identified by the SQI criteria
Full size table

See Table 2 for the results of a correlation analysis between h-index and SQI scores and the selected basic bibliometric parameters. All the presented correlation coefficient values demonstrate significant differences between SQI and h-index scores.

Table 2 Correlation analysis for the h-index and SQI scores
Full size table

The relations between h-index and SQI values for individual researchers are presented in Fig. 2. The h-index figures correlated significantly with the SQI values, r = 0.73, p < 0.0001.

Fig. 2
figure 2

Correlations between the h-index and SQI values

Full size image

Discussion

The current study aimed to carry out a pre-defined and structured appraisal of scientific outputs in a group of 480 top-ranked researchers, representing twelve medical areas, identified by selected key-words. The primary finding of the study concerned different ranking positions of the analyzed authors, classified by the h-index or by the SQI. Both individual ranking positions of particular investigators and the classification scores with respect to medical fields (see Fig. 1 and Table 1) reveal essential differences. The most impressive individual ranking promotion was identified for Snyder Shane A, an endocrinologist, who improved his position by 387 places when ranked by SQI (see Table 1). This author published 24 papers ‘only’ but all of them were cited and 22, out of 24, were cited ≥ 10 times and mean citation index per paper was 78.2. All that resulted merely in the 396th ranking position by the h-index and even the 9th ranking position by the SQI. Such a big discrepancy between those two, so different ranking scores clearly indicates that the quantitative and qualitative figures, which depict the scientific output of an individual investigator, may be totally unrelated. On the contrary, the scientist with the largest drop (− 249) in the SQI ranking system, when compared to the h-index ranking figures was the author, representing the field identified by the keyword ‘neoplasm’. This author published as many as 557 papers, out of which, 465 were cited. Interestingly enough, the author had the largest number of published papers in the whole analyzed group of 480 investigators. This author recorded 10,884 citations without self-citations and without the citations of all co-authors, his/her h-index score was 50 (the 74th ranking position), and the SQI was 63 (the 323rd ranking position.) It is a good example of an impressive output in terms of its quantity (more than one paper per week during 10 years), however, most of the published papers either found no citations or were cited less than 10 times (56.6%), and the average number of citations per paper was only 19.5. Such a person may then be depicted as someone with a very high publication performance which, however, does not translate into perception by scientific peers. Purely quantitative bibliometric indicators place the author very high in ranking, while the SQI reveals the rather ‘average’ quality of scientific work. That is why we consider SQI as the qualitative parameter mainly. Contrary, a simple citation count may not be a sufficient measure of the quality of the manuscript or author’s output, as demonstrated by Nieminen et al. (2006).

The influence of different factors, exerted on both indexes and identified in the analysis of correlation, seems to have been responsible for the observed differences between h-index and SQI figures (see Table 2). The SQI, much more than the h-index, concentrates on qualitative features in the scientific output of an individual investigator. The significance of pure quantitative factors, such as the number of publications, the number of citations, the number of cited papers and the number of papers cited ≥ 10 times, seems to be much less important in the SQI approach than it is in the h-index assessment. On the contrary, the factors more related to scientific quality than quantity, e.g. the percent of papers cited ≥ 10 times and the mean number of citations per paper are stronger related to the SQI than to the h-index.

Interestingly enough, the coefficient of correlation, calculated between the h-index and SQI as 0.73, is almost the same as that in the earlier study with 40 top-ranked researchers in the field of osteoporosis (r = 0.72) (Pluskiewicz et al. 2018). This observation may support the thesis of a repetitive relationship between the more qualitative SQI values and the more quantitative h-index figures in different study groups. On the other hand, it does not mean that the value of one index may be easily ‘predicted’ from the value of the second index. It could be possible when the correlation coefficient was > 0.9 but when it is around 0.7, we may still assume a broad range of factors which influence the final h-index and SQI results in different ways.

Moreover, the SQI has one unique feature, the bidirectional variability, which makes it distinctive from other bibliometric indices with only one-way, upward changes in their scores. For example, neither the number of citations nor the h-index scores can decrease, unlike the SQI—the figures of which may not only increase but also decrease if the quality of an individual scientific output gets any lower, as, indicated by the percent of papers, cited at least 10 times, and/or the mean citation per paper. In other words, an essential gain in the number of papers with low citation scoring will “dilute” the previously achieved SQI value. This feature of SQI should be considered as most important for longitudinal appraisal of the quality of research of an individual investigator. We plan to repeat our analysis for the same authors in future in order to identify changes and trace trends versus the present figures.

The present concept of SQI and our current study reveal some limitations though. Young authors with a small output cannot be adequately evaluated by this method. SQI is not able to take into consideration the position among the authors of a given paper (the first or a senior author). Reports from clinical trials, which are often not informative about the actual personal involvement of particular authors, cannot be removed from the Scopus database, leading to somewhat ‘overestimated’ SQI scores. The SQI is based on the mean citation scores, whereas, in case of some authors, their individual median of citations per paper could much better reflect their actual contribution. In the future, it would be interesting to develop an algorithm for ‘normalization’ of SQI scores among different science disciplines, to compensate for unequal chances to achieve high index values in some research areas.

In the current study, the SQI values were compared only with the h-index while additional comparisons with other indicators are to be considered, e.g. with the mean normalized citation score (MNCS) or the mean citation score. However, as those two indices have been previously criticized (Nieminen et al. 2006; Abramoa and D’Angelo 2016), we decided to compare the presented SQI with the perhaps most widely used bibliometric indicator (h-index) to make the presentation as clear as possible. The initial selection of the top 40 authors in each scientific area was based on the number of all their publications. In the subsequent step, the ranking position for each author was established by the h-index without auto-citations and citations of all co-authors. It means that some authors, not ranked within the initial list of top 40, could have had a higher h-index scores than the last subject in the list of 40 but they could not be taken into account. However, one should remember that the primary goal of the current study was to highlight the differences in the assessment of an individual author’s scientific output between h-index and SQI scoring modalities and not provide the personalized ranking lists.

Summing up, the scores from an assessment of the individual scientific output of 480 authors from 12 scientific areas were obtained in the current study, showing differences in the ranking positions of the authors between the h-index and the SQI estimates. Although strongly correlated, the h-index and SQI figures correspond to somewhat different aspects of individual scientific output. The SQI may be considered as a novel indicator of scientific output quality, whereas the h-index is stronger determined by quantitative measures (the number of papers and the number of citations). The ability to follow long-term changes in the quality of research output should be approached as a very important and unique feature, differentiating the SQI capacity from other bibliometric indices. We may thus approach the SQI as an interesting alternative in the assessment of individual scientific achievements.