Decision concordance with incomplete expert rankings in manufacturing applications

Abstract

The manufacturing field encompasses a number of problems in which some experts formulate their rankings of a set of objects, which should be aggregated into a collective judgment. For example, consider the aggregation of (1) the opinions of designers on alternative design concepts, (2) the opinions of reliability/safety engineers on the criticality of a set of failures, (3) the perceptions of a panel of customers on alternative aesthetic features of a product, etc. For these problems, Kendall’s concordance coefficient (W) can be used to express the degree of agreement between experts in a simple and practical way. Unfortunately, this indicator is applicable to complete rankings only, while experts often find it more practical to formulate incomplete rankings, e.g., identifying only the most/less relevant objects and/or deliberately excluding some of them, if they are not sufficiently relevant or well known. This research aims at extending the use of the traditional W to incomplete rankings, preserving its practical meaning and simplicity. In a nutshell, the proposed methodological approach associates a so-called “midrank” to all objects, even the ones that are not easily comparable with the other ones; subsequently, W can be applied to these midranks. The description is supported by several pedagogical examples.

Appendices

Appendix

1.1 Details on the reference scheme to determine midranks

This section provides some mathematical arguments to test the formulae contained in the scheme in Table 4. The first four sub-sections are dedicated, respectively, to (1) t-objects (ordered or not), (2) b-objects (ordered or not), (3) isolated objects and (4) remaining objects. The last sub-section provides a mathematical demonstration concerning the sum of the midranks of all the objects of a generic incomplete ranking.

1.1.1 t-Objects

In case the t-objects are ordered, let us assume that r is the rank of a generic t-object within a sub-ranking consisting of the t-objects only. If there are no isolated objects (i.e. objects intentionally excluded from expert evaluation), the rank of each t-object can only be r for any compatible complete ranking. In the presence of (k) isolated objects, the maximum possible rank of the generic t-object can be (r + k), in the hypothesis that all the isolated objects are placed ahead of it. In conclusion, we would have $R_{ij}^{\text{L}} = r$ and $R_{ij}^{\text{U}} = r + k$, with a corresponding midrank of $R_{ij} = \left( {R_{ij}^{\text{L}} + R_{ij}^{\text{U}} } \right)/2 = r + k/2$.

Let us now consider the case in which the t-objects are not ordered. In the best case, a generic t-object could be alone at the top of the complete compatible ranking, therefore $R_{ij}^{\text{L}} = 1$. In the worst case, in presence of (k) isolated objects, a generic t-object will be alone at the bottom of the sub-ranking consisting of the (t) t-objects and below the isolated objects themselves; it will, therefore, be in the position: $R_{ij}^{\text{U}} = t + k$. The corresponding midrank will, therefore, be $R_{ij} = \frac{{1 + \left( {t + k} \right)}}{2}$. In this case, the sum of the midranks of all the (t) t-objects would be $t \cdot \left( {\frac{1 + t + k}{2}} \right)$.

This result would be obtained also for ordered t-objects, in fact:

$$\sum\limits_{r = 1}^{t} {\left( {r + \frac{k}{2}} \right) = (1 + 2 + \cdots + t) + t \cdot \frac{k}{2} = \frac{{t \cdot \left( {t + 1} \right)}}{2} + t \cdot \frac{k}{2} = t \cdot \left( {\frac{1 + t + k}{2}} \right)} .$$

1.1.2 b-Objects

In case the b-objects are ordered, let us assume that s is the rank of a generic b-object within a sub-ranking consisting of the b-objects only. If there are no isolated objects, in any possible compatible complete ranking, the rank of each b-object can only be s plus the rank of the objects ahead of it, i.e., the (t) t-objects and the (n − t − b − k) remaining objects (see Sect. 3.2); the resulting rank will, therefore, be: s + t + (n − t − b − k) = s + n − b − k.

In the presence of (k) isolated objects, the maximum possible rank of the generic b-object can be (s + n − b − k) + k = s + n − b, in the hypothesis that all the (k) isolated objects are placed ahead of it. In conclusion, we would have $R_{ij}^{\text{L}} = s + n - b - k$ and $R_{ij}^{\text{U}} = s + n - b$, with a corresponding midrank of $R_{ij} = \left( {R_{ij}^{\text{L}} + R_{ij}^{\text{U}} } \right)/2 = s + n - b - k/2$.

Let us now consider the case in which the b-objects are not ordered. In the best case, a generic b-object could be alone at the top of the b-objects, therefore immediately behind the t-objects and the remaining objects; therefore, $R_{ij}^{\text{L}} = 1 + t + (n - t - b - k) = 1 + n - b - k$. In the worst case, in the presence of (k) isolated objects, a generic b-object will be alone at the bottom of a compatible complete ranking; it will, therefore, be in the position: $R_{ij}^{\text{U}} = n$. The corresponding midrank will, therefore, be: $R_{ij} = n + \frac{1 - b - k}{2}$. In this case, the sum of the midranks of all the (b) b-objects would be $b \cdot \left( {n + \frac{1 - b - k}{2}} \right)$.

The last result would also be obtained for ordered b-objects, in fact: $\sum\nolimits_{s = 1}^{b} {\left( {s + n - b - \frac{k}{2}} \right) = (1 + 2 + \cdots + t) + b \cdot \left( {n - b - \frac{k}{2}} \right) = \frac{{b \cdot \left( {b + 1} \right)}}{2} + b \cdot \left( {n - b - \frac{k}{2}} \right) = b \cdot \left( {n + \frac{1 - b - k}{2}} \right)}$.

1.1.3 Isolated objects

Being deliberately excluded as not sufficiently known, these objects could be placed in any position of the complete compatible rankings. In extreme cases, each of these isolated objects could then be placed:

• alone at the top, ahead of the other objects ($R_{ij}^{\text{L}} = 1$);

• alone in the bottom, below the other objects ($R_{ij}^{\text{U}} = n$).

The corresponding midrank will, therefore, be $R_{ij} = \frac{{R_{ij}^{\text{L}} + R_{ij}^{\text{U}} }}{2} = \frac{1 + n}{2}$. The sum of the midranks related to all the (k) isolated objects is $k \cdot \left( {\frac{1 + n}{2}} \right)$.

1.1.4 Remaining objects

These objects—which are not explicitly mentioned by the expert in the incomplete ranking—will be in an intermediate zone between the t-objects and the b-objects. Since these (n − t − b − k) objects are not ordered, the $R_{ij}^{\text{L}}$ and $R_{ij}^{\text{U}}$ values will be the same for all; the lowest possible rank will be the one immediately after the t-objects, i.e., $R_{ij}^{\text{L}} = t + 1$, while the highest possible rank will be the one immediately before the b-objects, i.e., $R_{ij}^{\text{L}} = n - b$. The corresponding midrank is $R_{ij} = \frac{{R_{ij}^{\text{L}} + R_{ij}^{\text{U}} }}{2} = \frac{{\left( {t + 1} \right) + \left( {n - b} \right)}}{2}$, while the sum of the midranks of all remaining objects is: $\left( {n - t - b - k} \right) \cdot \frac{{\left( {t + 1} \right) + \left( {n - b} \right)}}{2}.$

Proof concerning the sum of all object midranks

This sub-section contains a proof that the midranks in Table 4 are compatible with the convention adopted by Kendall for the calculation of W (in Sect. 2.1), i.e., that each row total of the rank table is equal to ${{n \cdot \left( {n + 1} \right)} \mathord{\left/ {\vphantom {{n \cdot \left( {n + 1} \right)} 2}} \right. \kern-0pt} 2}$. The proof is that adding the elements contained in the last column of Table 4 (“Midrank sum”), it can be obtained:

$$\begin{aligned} & t \cdot \left( {\frac{1 + t + k}{2}} \right) + b \cdot \left( {n + \frac{1 - b - k}{2}} \right) + k \cdot \left( {\frac{1 + n}{2}} \right) + \left( {n - t - b - k} \right) \cdot \frac{{\left( {t + 1} \right) + \left( {n - b} \right)}}{2} \\ & \quad = \frac{1}{2} \cdot (t + t^{2} + t \cdot k + 2 \cdot b \cdot n + b - b^{2} - b \cdot k + k + n \cdot k + n \cdot t + n + n^{2} - n \cdot b - t^{2} - t \\ & \quad + - n \cdot t + b \cdot t - b \cdot t - b - b \cdot n + b^{2} - t \cdot k - k - n \cdot k + b \cdot k )= \frac{1}{2} \cdot n \cdot \left( {n + 1} \right). \\ \end{aligned} .$$

(18)

2.1 Mind-expanding example

This section contains a further example with a double purpose:

• 1. Providing a preliminary assessment of whether the proposed procedure is based on legitimate assumptions and provides plausible results;

• 2. Justifying the need to integrate W with an additional uncertainty indicator (which will be developed in a future research).

Let us consider two decisional problems, both characterized by four experts (e₁–e₄) formulating individual rankings of four objects (o₁–o₄). In the first case, rankings are complete, while in the second case are incomplete. Figure 6 shows (a) these two sets of rankings and (b) their mutual compatibility (cf. definition of compatibility in Sect. 3.3). Additionally, Fig. 6c shows that both problems produce identical rank tables and, consequently, identical W values.

Fig. 6

Example of two decisional problems (one complete and one incomplete) characterized by identical rank tables and W values. The complete problem is compatible with the incomplete one

The suggested example brings out a somewhat questionable aspect, i.e., the procedure for calculating midranks actually equals objects with very different rank intervals. To clarify this concept, let us consider the (complete and incomplete) rankings by expert e₄ in Fig. 6. Regarding the complete ranking, the rank interval of o₄ is [2.5, 2.5] (absence of dispersion); regarding the incomplete ranking, it is [1, 4] (maximum possible dispersion for a rank interval related to a ranking with n = 4 objects). So, the proposed procedure synthesizes these two radically different rank intervals into the same midrank, i.e., 2.5. Is this result acceptable/reasonable?

Reflecting on the proposed procedure, the first part is a conventional transformation of each rank interval into a single equivalent rank (i.e., midrank). Of course, different conventional constructions could be adopted, resulting in different W values. This aspect raises a further question: Of all the possible ways of determining W for incomplete rankings, does the proposed one rely on reasonable hypotheses and provide plausible results? To provide a comprehensive answer to the above question, it would be necessary to carry out a structured study, as we plan to do in the future. Nevertheless, some preliminary arguments to support the proposed technique are presented below, in the form of comments about the results of the previous example.

Let us consider a rigorous but also very laborious way of determining W for problems with incomplete rankings; this method is based on three steps (De Loof et al. 2011; Bruggemann and Carlsen 2011):

• For each incomplete ranking, all the possible compatible complete rankings are generated.

• Combining the above complete rankings, all the possible complete problems that are compatible with the initial incomplete one are identified.

• The W value related to each complete problem is determined. Then the distribution of the resulting W values is constructed and studied.

Table 7 exemplifies this exercise for the four incomplete rankings in Fig. 6(2). Even very simple incomplete rankings may generate a relatively large number of compatible complete rankings, e.g., the one formulated by expert e₃ generates thirteen complete rankings. In fact, this problem can be classified as NP-hard, as its complexity increases exponentially with the number of objects and experts (Bruggemann and Carlsen 2011).

Table 7 Possible complete rankings, which are compatible with the four incomplete rankings at the top

Considering the set of rankings compatible with a certain incomplete ranking, it is interesting to examine the rank distributions. For example, Fig. 7a shows the rank distributions related to the thirteen rankings compatible with the incomplete ranking by expert e₃. Analyzing these distributions reveals some interesting aspects:

Fig. 7

Ranks and corresponding distributions related to the four objects (o₁–o₄), considering the thirteen complete rankings compatible with the incomplete ranking by expert e₃ (see Table 7)

• 1. All four distributions, respectively, referred to each of the four objects (o₁–o₄), are symmetrical. This symmetry is probably related to the structure of the incomplete rankings considered in this specific case (Caperna and Boccuzzo 2018). In the future, we plan to assess the legitimacy of the symmetry hypothesis in more rigorous and general terms.

• 2. Due to the aforesaid symmetry, the average value of each distribution coincides with the midrank (cf. Figs. 6c, 7).

• 3. The distributions of the object ranks are generally correlated with each other, as also exemplified in the Pearson correlation matrix in Table 8.

  Table 8 Pearson correlation table of the ranks related to the four objects (o₁–o₄), considering the thirteen complete rankings compatible with the incomplete ranking by expert e₃ (see Table 7)

Subsequently, we consider all the possible combinations between the sets of compatible complete rankings in Table 7, i.e., (1) the three complete rankings related to the incomplete rankings by e₁, (2) the three ones related to the incomplete rankings by e₂, (3) the thirteen ones related to the incomplete rankings by e₃, and (4) three ones related to the incomplete rankings by e₄. Consequently, 3·3·13·3 = 351 complete decisional problems, which are compatible with the initial (incomplete) one, can be identified. For each of these problems, it is then possible to determine a corresponding value of W (through the traditional procedure in Sect. 2) and then to study the corresponding W distribution. We point out that these 351 complete problems arise from an incomplete problem with a relatively small number of experts and objects. This denotes the low sustainability of the proposed construction, for realistic problems characterized by a large number of objects and/or experts.

The histogram in Fig. 8 represents the distribution of the W values resulting from the example. We note that this distribution is slightly right-skewed (median below mean value) and has a relatively high dispersion (standard deviation of 19.3%). The mean value of W is equal to 39.7%. Interestingly, the W value determined through the procedure based on midranks (i.e., 44.8%, as shown in Fig. 6) is relatively close to the previous mean value, denoting a certain plausibility of the procedure itself.

Fig. 8

Histogram and descriptive statistics of the W distribution related to the 3·3·13·3 = 351 possible complete problems examined

The authors are currently developing a new technique to determine an indicator that expresses the uncertainty of the W values, calculated using midranks; this indicator should somehow take into account the dispersion of the initial rank-intervals, without neglecting any correlations between them. In addition, this new indicator could be interpreted as a proxy for the dispersion of the above-exemplified W distribution, which avoids to carry out such a laborious construction.

2.2 Further data concerning the six problems exemplified

Tables 9, 10, 11, 12, 13, 14 illustrate the rankings and descriptive parameters (n, t, b, k, c and $\bar{c}$) related to the six problems exemplified in Sect. 3.4. For each ranking, the (k) isolated objects are specified in brackets; “(o)” denotes t/b-objects in case they are ordered, while “(u)” in case they are unordered.

Table 9 Complete problem (0) and respective descriptive parameters

Table 10 First incomplete problem (1) and respective descriptive parameters

Table 11 Second incomplete problem (2) and respective descriptive parameters

Table 12 Third incomplete problem (3) and respective descriptive parameters

Table 13 Fourth incomplete problem (4) and respective descriptive parameters

Table 14 Fifth incomplete problem (5) and respective descriptive parameters

Tables 15, 16, 17, 18, 19 and 20 contain the rank tables relating to the six problems (0–5) shown in Tables 9, 10, 11, 12, 13 and 14, respectively.

Table 15 Rank table concerning the complete problem (0) in Table 9

Table 16 Rank table concerning the first incomplete problem (1) in Table 10

Table 17 Rank table concerning the second incomplete problem (2) in Table 11

Table 18 Rank table concerning the third incomplete problem (3) in Table 12

Table 19 Rank table concerning the fourth incomplete problem (4) in Table 13

Table 20 Rank table concerning the fifth incomplete problem (5) in Table 14