A heuristic rating estimation algorithm for the pairwise comparisons method
Abstract
The pairwise comparisons method is a powerful tool used for establishing the relative order between different concepts in situations in which it is difficult (or sometimes even impossible) to provide explicit rating. Appropriate ratings are determined by solving eigenvalue problem for the pairwise comparisons matrix. This study presents a new iterative heuristic rating estimation algorithm that tries to deal with the situation when exact estimations for some concepts (stimulus) \(C_{K}\) are a priori known and fixed, whilst the estimates for the others (unknown concepts \(C_{U}\)) need to be computed. The relationship between the local estimation error, understood as the average absolute error \(E(c)\) over all direct estimates for the concept \(c\in C_{U}\) and the pairwise comparisons matrix inconsistency index is shown. The problem of convergence of subsequent intermediate results is discussed and the convergence conditions are given.
Keywords
Decision analysis Pairwise comparisons Iterative algorithms Data inconsistency1 Introduction
The pairwise comparisons (PC) method was introduced in its early form by Fechner (1966), then it was popularized and developed by Thurstone (1994). The introduction of hierarchical structures by Saaty (2008) was another important contribution to the PC method, providing the methodology and practical ways to deal with the large amounts of criterion parameters. Initially the PC method was used in the scientific study of psychometrics and psychophysic (Thurstone 1994), however, it then came to be used in other areas of applications, such as complex decision theory (Saaty 2008), economics (Peterson and Brown 1998), voting systems (Tideman 1987) and others. Since the data, which are input to the PC method are the result of human judgment, it is very easy for inaccuracy to occur. Hence, the input data set is frequently ambiguous and does not allow users to draw firm conclusions. There are several indexes of the data inconsistency (Bozóki and Rapcsak 2008), including the best known Saaty’s eigenvector method (Saaty 1980), Least Squares Method, Chi Squares Method, Koczkodaj’s distance based inconsistency index (Koczkodaj 1993), and others. Using these indexes, the reliability of the data can be assessed, hence, the confidence in the result can be evaluated. The answer to the question of how much input data must be consistent to ensure the result reliability is the subject of empirical research. For instance, according to Saaty’s recommendation every occurrence of the consistency ratio greater than or equal to \(0.1\) should be the subject of reexamination of the pairwise judgments until the inconsistency becomes less than or equal to the desired value (Bozóki and Rapcsak 2008; Triantaphyllou et al. 1990). The main criticism of this approach relates to its separation from the data and lack of localizing the most problematic matrix elements (Bozóki and Rapcsak 2008; Koczkodaj 1993; Triantaphyllou et al. 1990). In contrast, Koczkodaj’s inconsistency index has a meaningful interpretation and provides information about the inconsistency location, but it does not provide an exact answer to the question of how good the average data sample is. Inconsistency identified as too high, must be reduced to an acceptable level (ideally to zero). Since the ratio coefficients, which are the input to the PC method, frequently represent experts’ judgements, thus a natural way of inconsistency reduction is to call the expert panel once again and ask the professionals gathered to agree on the opinion (Gomes 1993). Because such a solution is usually timeconsuming and expensive, heuristic algorithms of inconsistency reduction have been proposed (Koczkodaj and Orłowski 1999; Koczkodaj and Szarek 2010; Gomes 1993; Xu and Wei 1999; Bozóki 2008; Temesi 2006; Cao et al. 2008). The result of these algorithms is a new set of data, which preserves most of the decision maker’s original judgment structure and significantly reduces the data inconsistency.
The proposed innovative solution approaches the problem differently. It does not attempt to minimize inconsistency in the data, but rather proposes a way of using the data, which takes into account their inconsistency. Hence, knowing the exact values for a few concepts^{1} and some inconsistent set of ratios between them, the data analyst is able to estimate values of all other concepts with errors depending on the degree of data inconsistency. The presented approach is comparable to the inconsistency reduction methods mentioned above, since the set of concepts for which the estimates are known can be easily transformed into a consistent set of data (as addressed in Sect. 8).
The first and second sections of the article focus on the presentation of the necessary facts and definitions concerning the pairwise comparisons methods. Section three formulates formally the problem considered in the work. The fourth and fifth, sections provide the Heuristic Rating Estimation (HRE) algorithm together with a numerical example demonstrating the algorithm application in practice. The sixth section shows the relationship between errors of estimations obtained by using the HRE algorithm and the data inconsistency. The next, seventh section addresses the case of which the subsequent estimation sets converge to some limit. Finally, the last two sections contain the closing discussion and summary.
2 A pairwise comparisons method
A crucial part of the PC method is \(M=(m_{ij})\wedge m_{i,j}\in \mathbb R _{+}\wedge i,j\in \{1,\ldots ,n\}\) a PC matrix that expresses some quantitative relation \(R\) over the finite set of concepts \(C\overset{df}{=}\{c_{i}\in \fancyscript{C}\wedge i\in \{1,\ldots ,n\}\}\) where \(\fancyscript{C}\) is a non empty universe of concepts and \(R(c_{i},c_{j})=m_{ij},\, R(c_{j},c_{i})=m_{ji}\). Traditionally, these concepts are interpreted as subjective stimuli (Thurstone 1994), whilst the values \(m_{ij}\) and \(m_{ji}\) are considered as the relative importance indicators (stimulus intensities), so that according to the best knowledge of an expert the significance of \(c_{i}\) equals \(m_{ij}c_{j}\).
Definition 1
A matrix \(M\) is said to be reciprocal if \(\forall i,j\in \{1,\ldots ,n\}:m_{ij}=\frac{1}{m_{ji}}\), and \(M\) is said to be consistent if \(\forall i,j,k\in \{1,\ldots ,n\}:m_{ij}\cdot m_{jk}\cdot m_{ki}=1\).
Since the knowledge stored in the PC matrix usually comes from different professionals in the field of relation \(R\), it often results in inaccuracy. In such a case, reasoning using the data gathered in \(M\) may give ambiguous results. This observation gave rise to the research on the concept of data consistency. In the ideal case, \(M\) is consistent, and there is no doubt as regards the value assigned to the concept \(c_{j}\) if the value assigned to \(c_{i}\) and \(m_{ji}\) is known. Unfortunately, in practice the knowledge in \(M\) is inconsistent, and professionals using the PC method have to deal with this incoherence. Thus, it is important to answer the question of how inconsistent the knowledge in the PC matrix is. There are a number of inconsistency indexes, including Eigenvecor Method (Saaty 2008), Least Squares Method, Chi Squares Method (Bozóki and Rapcsak 2008), Koczkodaj’s distance based inconsistency index (Koczkodaj 1993) and others. For the purpose of this article, the Koczkodaj’s distance based inconsistency index has been adopted since it is the only localizing index amongst the above mentioned indexes.
Definition 2
Intuitively speaking, since in an “ideal” matrix \(\forall i,j,k\in \{1,\ldots ,n\}:m_{ij}\cdot m_{jk}\cdot m_{ki}=1\) the Koczkodaj’s index localizes the worse triad (by the Euclidean distance) from this ideal.
PC matrices may be overcomplete, i.e. there is more than one estimation describing one and the same relation between \(v_{i}\) and \(v_{j}\), but they can be also incomplete, i.e. not all values \(m_{ji}\) are defined. While the first situation can be addressed in many ways, for instance the estimates related to the same pair of concepts can be averaged, the other is not desirable. It indicates that the model is lacking of knowledge, although the missing estimates can be somehow compensated (for example, by employing the properties of reciprocity, consistency or transitivity) (Koczkodaj et al. 1999). Hence, for further consideration, it will be assumed that the \(n\times n\) PC matrix \(M\) is complete in the sense specified below.
Definition 3
A matrix \(M\) \((n\times n)\) is said to be complete if \(\forall i,j=1,\ldots ,n:m_{ij}\) is defined.
Due to the further consideration it is useful to define a graph structure over the set \(C\) and matrix \(\mathsf M \).
Definition 4

\(C\), is a set of vertices

\(E\subseteq C\times C\) so that \((v_{i},v_{j})=e\in E\Leftrightarrow m_{ji}\) exists, is a set of edges

\(M:E\rightarrow \mathbb R \) so that \(M(e)\overset{df}{=}m_{ji}\wedge e=(v_{i},v_{j})\) is a function of experts’ assessments
Wherever it does not raise doubts, instead of \(M(e)\wedge e=(u,v)\) the function \(M\) will be written with two arguments i.e. \(M(u,v)\).
3 Problem formulation
Since the concepts are linked to each other by a quantitative relation \(R\), then assuming that the exact values of some concepts are known, the values of the others should be proposed. Thus, let \(\mu :C\nrightarrow \mathbb R _{+}\) be a partial function that assigns to some concepts from \(C\subset \fancyscript{C}\) positive values from \(\mathbb R _{+}\). Hence, the concepts for which the actual value \(\mu \) is known are denoted by \(C_{K}\subset C\) and called known concepts, whilst concepts for which \(\mu \) need to be determined are denoted \(C_{U}=C\backslash C_{K}\) and called unknown concepts. The relation between different concepts in terms of the function \(\mu \) is represented in the form of the PC matrix \(M\), so that \(m_{ji}\mu (v_{i})=\mu (v_{j})\). Since \(m_{ji}\) usually aims to express how much greater or smaller \(v_{i}\) is than \(v_{j}\) with respect to \(\mu \) by convention it is assumed that elements of the \(n\times n\) PC matrix are positive real numbers, i.e. \(m_{ji}\in \mathbb R _{+}\) where \(i,j\in 1,\ldots ,n\).
4 Heuristic rating estimation algorithm
The principle of operation of the rating estimation algorithm (Listing 1) is to iteratively assign the value \(\mu (u)\) to every unknown vertex \(u\in C_{U}\) by calculating the mean of its samples, and then choosing from among all the calculated estimations the one which is optimal. The idea of the procedure comes from the BFS algorithm (Cormen et al. 2009), which, layer by layer, traverses the graph of interest. In the presented approach, the estimates for the next layer are computed on the base of the previous layer, assuming that vertex repetition in different steps is allowed. The algorithm stops when either the appropriate layer is reached or if the algorithm converges for the given \(G\), then it stops when the distance in the sense of the chosen metrics (see Eq. 47) between the subsequent estimates for elements in \(C_{U}\) is smaller than some desired \(\epsilon \). If the algorithm does not converge (when \(G\) is fixed) then the number of steps that need to be made is explicitly set at the beginning of the estimation procedure. In the case of standard complete PC matrices the graph is a directed clique, where every single vertex is connected to each other. In such a case it appears that even in the first step all the vertices are visited, whilst in the second step the computed estimations may take into account all possible values^{2} gathered in \(M\) except those describing ratios between elements in \(C_{K}\). Thus, when the algorithm is not convergent, a few iterations of the HRE procedure seem to be a reasonable choice.
5 Numerical example
The matrix \(M\) is used for the generation graph \(G\), the set of known concepts \(C_{K}\) is formed by the projects \(u_{4}\) and \(u_{5}\), and finally the unknown concept set is \(C_{U}=\{u_{1},u_{2},u_{3}\}\). Since all concepts are reachable in the first step, and during the second step the \(\mu \) mapping is computed for every \(c\in C_{U}\) using all other concepts in \(C\), then for the practical demonstration of the algorithm the number of levels traversed is limited to \(2\). In fact, for the matrix \(M\) as given (Eq. 5) the subsequent estimations calculated by HRE converge. Thus, instead of limiting the number of steps, an appropriately small \(\epsilon \) can be chosen. This case will be discussed later, after the convergence criterion for HRE is defined.
6 Data consistency and estimation error
For any algorithm, the result of which being inaccurate, a natural questions concern the accuracy of the resulting solution, and what should be done to improve the accuracy. The presented procedure is based on the fundamental sample mean estimation equation (Walpole 2012), where for the purpose of the algorithm the product \(\mu (u_{i})\cdot M(u_{i},u_{j})\) is treated as a sample, whilst \(\mu (u_{j})\) denotes the expected value inferred from samples. In this case the natural measure of the algorithm output accuracy is an estimation error understood as the distance between sample and the mean. Of course, the smaller the error, the more accurate the result. According to the popular adage “garbage in, garbage out” it is expected that even the best algorithm is not able to provide good output if the input data are bad. Hence, it might be expected that the estimation errors of HRE depend on data consistency, and are smaller in correlation with data inconsistency. The reasoning below supports this assertion.
Theorem 1
Proof
7 Convergence of solution
One of the immediate questions to come up is about the optimal number of iterations. The answer is not obvious, since the result of the algorithm depends on the input data. In particular, it is easy to construct simple graphs over the inconsistent PC matrix \(M\) in cases where every further step of the algorithm significantly increases the absolute error of estimation. For instance, the graph \(G\) such that \(G=(C_{K}\cup C_{U},E,M)\) where \(C_{K}=v_{1},C_{U}=v_{2},v_{3}\) and \(\{(v_{1},v_{2}),(v_{1},v_{3}),(v_{2},v_{3}),(v_{3},v_{2})\}\subseteq E\) where the values \(M(v_{1},v_{2}),\, M(v_{1},v_{3}),\,M(v_{2},v_{3}),\, M(v_{3},v_{2})\) or all are less than or greater than \(1\). In the first case the subsequent estimates \(\mu (v_{2})\) and \(\mu (v_{3})\) tend to \(0\), and reversely, if all the values are greater than \(1\), then \(\mu (v_{2})\) and \(\mu (v_{3})\) tends to \(\infty \). In the general case, the inequality 37 (Th. 1) suggests that with every subsequent step errors may increase. This leads to the conclusion that the optimal strategy is “the fewer steps the better”. Therefore, if the algorithm is not convergent, traversing at most one or two levels seems like a good idea.
 1.
The set \(C_{U}\) is relatively small (\(C_{K}\) is relatively large),
 2.
The estimated values \(\mu (v)\) are similar.
The convergence of the algorithm implies the convergence of the estimation error. Unfortunately, the limit towards which the estimation error tends is not necessarily the smallest possible error value. Hence, in order to minimize the error, the user needs to choose the best estimation from all the estimations generated during the course of the algorithm (Listing: 1, line: 21).
In the case of the previously considered example (Sect. 5) all the three conditions \(Q_{1},Q_{2}\) and \(Q_{3}\) are below 1. Thus, the algorithm is convergent and the computed limits for \(\mu (v_{1}),\mu (v_{2})\) and \(\mu (v_{3})\) are 2.758, 4.578 and 3.493, respectively. The absolute average estimation errors converge to \(e_{\mu }(v_{1})=0.12,\, e_{\mu }(v_{2})=0.631\) and \(e_{\mu }(v_{3})=0.338\), and are minimal with respect to the mean of errors Open image in new window .
8 Discussion
The idea underlying the HRE algorithm is the assumption that experts work independently and try to do their job as best as they can. Thus, they may be wrong in a random manner, and if so, it makes sense to accept their estimates as a part of samples, and the expected value of the sample (the arithmetic mean) as the alleged value of the estimated \(\mu \) for the unknown concept. The absolute average estimation error may indicate how good such an estimation is. Depending on the data, the algorithm may or may not converge. If the algorithm does not converge (subsequent estimates are getting bigger) or converges to \(0\), selecting from among the first few estimates the one with the smallest error seems to be the best choice. If the algorithm converges to a non \(0\) value, it is very often useful to generate subsequent estimates until the current estimate does not differ from the limit (fixed point) with some \(\epsilon \), then to choose a set of estimations (among those generated) for which the average estimation error is minimal. Of course, this is not the only possible strategy. For instance, if some subset of \(C_{U}\) is particularly important, only errors for its elements may be taken into account while determining the optimal set of estimations.
Although the presented considerations in (Sect. 6 and 7) assume that the matrix \(M\) is complete (see Def. 3), the HRE algorithm seems to work without this assumption. For incomplete matrices, the value \(\mu (u)\) for \(u\in C_{U}\) can be determined as long as a path exists in the pairwise comparisons graph \(G\) over \(M\) (Def. 4) between \(u\) and some element \(v\in C_{K}\). Hence, there is a chance that the HRE algorithm may support different pairwise comparisons techniques also when the input data are incomplete. In particular, it might be useful for the AHP (Saaty 1977) approach. The properties of the presented algorithm for the incomplete sets of input data will be the subject of future research.
9 Summary
The HRE algorithm presented here for computing estimations of initially unknown concepts \(C_{U}\) using information about known concepts \(C_{K}\) and the matrix \(M\) proposes a new approach to the pairwise comparisons method. It defines the intuitive algorithm of using the pairwise comparisons matrix \(M\) for determining the most probable values of unknown concepts (original stimulus) on the basis of the known concepts. The presented procedure iteratively generates sets of estimations, then chooses the set which has the smallest average absolute mean estimation error. According to the proven theorem, the size of the estimation error depends on \(\fancyscript{K}\)—Koczkodaj’s distance based inconsistency index shows that the lower the inconsistency index, the lower the estimation errors. The theorem can be particularly useful when the number of iterations of the HRE algorithm is small. In such cases, it may in practice be used to estimate the size of the estimation errors.
For some input data sets, the subsequent estimations produced by the HRE algorithm converges. If this happens, the estimation errors also converge to some limit, thus the number of estimation sets produced by the HRE algorithm does not need to be limited to a few elements. In such a case the sets of estimations can be generated until the limit towards which the HRE results converge will not be close enough. Such a situation has also been addressed in the paper. The given conditions of convergence have an intuitive explanation and in many practical situations are likely to be met.
The HRE algorithm is suitable for any matrix with positive elements, i.e., even in those situations where the applicability of the classical eigenvector method can be limited (finding the largest absolute eigenvalue of a nonreciprocal matrix may be difficult). The presented algorithm remains open for much more data than can be stored in a single pairwise comparisons matrix. Due to the graph representation of the problem, multiple values defining ratios between the same pairs of concepts can be easily encompassed within the algorithm as multiple arcs between the same pairs of vertices.
Footnotes
 1.
Thurstone (1994) originally called them stimulus.
 2.
Following the reciprocity principle all the \(M\) values in the form \(M(c_{i},c_{j})\) where \(c_{i}\in C_{U}\wedge c_{j}\in C_{K}\) can be used to create expressions \(\frac{\mu (c_{j})}{M(c_{i},c_{j})}\) are treated as just one specimen of \(\mu (c_{i})\).
Notes
Acknowledgments
The author would like to thank Prof. W.W. Koczkodaj, Laurentian University, Sudbury, Ontario, Canada, for the inspiration and valuable Skype discussions about the pairwise comparisons method. The author is also grateful to the anonymous reviewers for their comments and suggestions which have helped improve the quality and content of the paper.
References
 Bozóki S (2008) Solution of the least squares method problem of pairwise comparison matrices. Central European Journal of Operations Research 16(4):345–358CrossRefGoogle Scholar
 Bozóki S, Rapcsak T (2008) On Saaty’s and Koczkodaj’s inconsistencies of pairwise comparison matrices. Journal of Global Optimization 42(2):157–175CrossRefGoogle Scholar
 Bronstein IN, Semendjajew KA, Musiol G, Mühlig H (2005) Handbook of mathematics, 5th edn. Springer, Frankfurt am MainGoogle Scholar
 Cao D, Leung LC, Law JS (2008) Modifying inconsistent comparison matrix in analytic hierarchy process: a heuristic approach. Decis Support Syst 44(4):944–953CrossRefGoogle Scholar
 Cormen TH, Leiserson CE, Rivest RL, Stein C (2009) Introduction to algorithms, 3rd edn. MIT Press, CambridgeGoogle Scholar
 Fechner GT (1966) Elements of psychophysics, vol 1. Holt, Rinehart and Winston, New YorkGoogle Scholar
 Gomes L (1993) Efficient reduction of inconsistency in pairwise comparison matrices. Syst Anal Model Simul 11(4):333–335Google Scholar
 Koczkodaj WW (1993) A new definition of consistency of pairwise comparisons. Math Comput Model 18(7):79–84CrossRefGoogle Scholar
 Koczkodaj WW, Herman MW, Orlowski M (1999) Managing null entries in pairwise comparisons. Knowl Inf Syst 1(1):119–125CrossRefGoogle Scholar
 Koczkodaj WW, Orłowski M (1999) Computing a consistent approximation to a generalized pairwise comparisons matrix. Comput Math Appl 37(3):79–85CrossRefGoogle Scholar
 Koczkodaj WW, Szarek SJ (2010) On distancebased inconsistency reduction algorithms for pairwise comparisons. Log J IGPL 18(6):859–869CrossRefGoogle Scholar
 Peterson GL, Brown TC (1998) Economic valuation by the method of paired comparison, with emphasis on evaluation of the transitivity axiom. Land Econ 74:240–261CrossRefGoogle Scholar
 Saaty TL (1977) A scaling method for priorities in hierarchical structures. J Math Psychol 15(3):234–281CrossRefGoogle Scholar
 Saaty TL (1980) The analytic hierarchy process: planning, priority setting, resource allocation. McGrawHill International Book, New YorkGoogle Scholar
 Saaty TL (2008) Relative measurement and its generalization in decision making. Why pairwise comparisons are central in mathematics for the measurement of intangible factors. The analytic hierarchy/network process. Stat Oper Res (RACSAM) 102:251–318Google Scholar
 Temesi J (2006) Consistency of the decisionmaker in pairwise comparisons. Int J Manag Decis Mak 7(2):267–274Google Scholar
 Thurstone LL (1994) A law of comparative judgment, reprint of an original work published in 1927. Psychol Rev 101:266–270CrossRefGoogle Scholar
 Tideman TN (1987) Independence of clones as a criterion for voting rules. Soc Choice Welf 4:185–206CrossRefGoogle Scholar
 Triantaphyllou E, Pardalos PM, Mann SH (1990) A minimization approach to membership evaluation in fuzzy sets and error analysis. J Optim Theory Appl 66(2):275–287CrossRefGoogle Scholar
 Walpole RE et al (2012) Probability & statistics for engineers & scientists, 9th edn. Prentice Hall, Englewood CliffsGoogle Scholar
 Xu Z, Wei C (1999) A consistency improving method in the analytic hierarchy process. Eur J Oper Res 116(2):443–449CrossRefGoogle Scholar
Copyright information
Open AccessThis article is distributed under the terms of the Creative Commons Attribution License which permits any use, distribution, and reproduction in any medium, provided the original author(s) and the source are credited.