Consistent Construction of Evaluation Threshold Values and Rules for Heterogeneous Linguistic Input Information

Abstract

This study firstly proposes a simpler method for evaluating one certain object’s quality with multiple criteria according to some preset evaluation threshold values that are real numbers. In real life, numerous individual valuations are provided with distributional linguistic input information and with multiple criteria, and thus they can become heterogeneous. Against this background, by using OWA weight functions we propose an extended setting and some methods to generate distributional evaluation threshold values which are suitable for the corresponding thresholds-based evaluation method. Some special definitions and formulations are also well provided with necessary analyses and comments. A numerical example of reservoir evaluation and effect are also illustrated.

Introduction

Numerous evaluation problems are based on quantitative analysis and its related decision taking [1,2,3,4]. In most of those problems, to judge the evaluation objects involves several evaluation criteria rather than a single criterion. Therefore, merging those collected individual information from different criteria into a single one can significantly facilitate the corresponding overall evaluation and decision making.

As commonly known, multi-criteria decision making (MCDM) [5,6,7,8,9] is a widely used evaluation and decision making tool which considers numerical inputs (generally real numbers) and applies an elicited weight vector to perform weighted average (WA) over the inputs, merging them and then yielding a single evaluation value which is still a real number. If there are several evaluation objects (belonging to a same kind) under evaluation and comparison, then by the returned outcomes using MCDM or other evaluation methods, we can select one or some few objects as the desired ones having better properties and outcomes. If there is only one evaluation object under consideration, then in general decision makers may need to judge whether the object is qualified by comparing its evaluation outcome with a predetermined threshold value, which is actually still a decision making problem.

Recall that aggregation operators [10, 11] have been widely applied in evaluation and information fusion, and the related theories have been fast developed during the last few decades [11,12,13,14,15]. A type of well-known aggregation operators used in MCDM include weighted average, geometrical weighted average, and weighted harmonic mean [10], etc. A common feature in those operators is that they all apply a well determined normalized weight vector (or weight function). Note that with such a weight function but performed in a very different order of inputs positions, Yager proposed another aggregation scheme called ordered weighted averaging (OWA) operators [16], which has been widely applied in considerably many areas [17,18,19]. It is also noteworthy that the OWA operators can be regarded as a generalization of order statistic (OS) [10] and can also be understood as a special case of the well-known Choquet Integrals with symmetrical capacities (note also that WA operators can be seen as the Choquet Integrals with additive capacities) [10].

All of the above-discussed operators take real numbers as inputs and yield real number output for further decision taking and evaluation. However, in a myriad of evaluation practices, the quantities provided and collected by experts and respondents, and the overall evaluation results for decision makers to take further judgment, are expressed by linguistic information. Researchers studied and proposed some linguistic information and aggregation models [20,21,22]. The linguistic information considered in this study is a very common one which is much easier to be obtained and collected via different approaches such as by direct inquiring with experts or customers, internet questionnaires, and group meeting and voting.

Put simply, in real evaluation and decision making problems, sometimes the collected input data are not always real values which can be easily transformed or standardized into unit interval [0,1]. For example, some individual evaluations are usually obtained by a familiar linguistic evaluation based on a linearly ordered set $H^{(r)} = (\{ 1,...,r\} ,\underline { \prec } )$ (later sometimes we may only consider $\{ 1,...,r\}$ and neglect its associated order relation $\underline { \prec }$ which will not make any confusion arise). Such linearly ordered set can be embodied or realized by some linguistic term set such as ({1 “excellent”, 2 “good”, 3 “average”, 4 “substandard”},$\underline { \prec }$) in which $\underline { \prec }$ indicates a preference relation to show that linguistic evaluation term i is “better” than term j whenever $i < j$. Another instance of this type of evaluation information is ({1 “recommended”, 2 “satisfied”, 3 “unqualified”}, $\underline { \prec }$).

With one such linguistic term set, an expert, stakeholder or customer of a certain evaluation problem can provide his/her own judgment over some evaluation object (like a product or a type of service). For example, a customer can be inquired about the quality of a product and offer only one linguistic term as his/her evaluation, say, “good” or “average.” Note that such survey can be carried out by involving multiple persons rather than only one single evaluation subject like that customer. Therefore, with simple statistics, a normalized distribution can be naturally obtained over the linguistic term set $H^{(r)}$, which, in this study using some conventional way, is expressed by a nonnegative function $p:H^{(r)} \to [0,1]$ satisfying $\sum\nolimits_{i = 1}^{r} {p(i)} = 1$.

Since the forgoing mentioned linguistic information is more complex in structure than the simple real number, and for different involved criteria there need differently designed linguistic term sets for making surveys, then the handling, judgment and possible merging of them need special techniques. The pervasiveness of such type of linguistic information in real decision making problems and the feasibility for collecting and dealing with such linguistic information, make the corresponding studies important and meaningful. This study will discuss some special and relevant automatic judgment and evaluation techniques which are mainly based on some well-designed partitioning and thresholds determination rules. One of the advantages of this study lies in that it can provide some relatively objective evaluation scheme within the complex and subjectivity permeated comprehensive decisional scenarios. The theoretical value of the study can be also found in the area of computational intelligence and aggregation theories.

The study will also revolve around some concepts and methods of OWA weight functions’ defining and determination. In addition, we need to make a special note that the OWA weight functions are normally closely associated with OWA operators, but in this study they will be independently used in different decision scenarios and no longer be linked with OWA operators for taking corresponding aggregations.

The remainder of this study is organized as follows. Section 2 firstly formulates a simpler partitioning method for real valued inputs and then presents necessary preparations for later discussions. Section 3 elaborates a comprehensive evaluation method using linguistic evaluation thresholds for heterogeneous inputs. Section 4 provides an application in the evaluation of reservoir operation quality and effect. Section 5 concludes and remarks this study.

Some Preparation for the Evaluation Method with Heterogeneous Evaluation Information

In this section, we firstly formulate a relatively simpler evaluation method using partitioning method with real valued inputs, and then some necessary review, definitions and comments are prepared for the later discussed method for heterogeneous evaluation information.

The Formulation of the Evaluation Method Under Real Valued Inputs

Some terminologies and expressions are fixed in what follows. The normal numerical input information (for evaluation) is defined as a nonnegative bounded real function $x:\{ 1,...,n\} \to [0,1]$ ($n \ge 2$). The collection of all such nonnegative bounded real functions defined on $\{ 1,...,n\}$ is conventionally denoted by $[0,1]^{n}$.

With the above information, we next design an evaluation method based on given threshold values and summarize it into the following procedures. One may observe that only few human interventions are involved, which can provide more objectivity and efficiency in some real decision making and evaluation problems. The method is suitable for the situation where we only need to decide a linguistic evaluation value for one certain object under evaluation. Nevertheless, it is noteworthy that the following model might become unsuitable for the decision situation where it is needed to compare several alternative evaluation objects and select only one optimal object. This is because the method is mainly based on qualitative evaluation, and thus often two or more objects may have a same evaluation (such as qualified or unqualified). Besides, quantitative evaluations usually are sensitive, and they may not be very suitable for comparing individual evaluation values which cannot be commensurable.

Remark

The choices of evaluation thresholds $(a,b)$ may influence the final decision making results. In practices, several different experts can be invited to determine their individual suggested thresholds $(a_{i} ,b_{i} )$ and then apply an average of those thresholds.

Some Definitions for Dealing with Heterogeneous Evaluation Information

As mentioned in Introduction, the individual evaluation information collected for some certain criteria can be with the form of a nonnegative function $p:H^{(r)} \to [0,1]$ satisfying $\sum\nolimits_{i = 1}^{r} {p(i)} = 1$. For different criteria $\{ C_{i} \}_{i = 1}^{n}$, it may apply some different linguistic term sets $H^{(r)} = (\{ {1},...,r\} ,\underline { \prec } )$ with dimension r varying in $\{ 2,3,...\}$. Since those different linguistic term sets are heterogeneous and non-commensurable, then for better formulation and convenient analysis, we should design a set of strict definitions and concepts for further formulating purpose.

We firstly review, rephrase or redefine some basic concepts relating to OWA weight vectors which will serve as the main ingredients in the discussed methods in this study.

Definition 2.1

[16] In this study, a weight function defined on the linearly ordered set $H^{(r)} = (\{ 1,...,r\} ,\underline { \prec } )$, $w^{(r)} :\{ 1,...,r\} \to [0,1]$, is called an OWA weight function with dimension r. The set of all r-dimensional OWA weight functions is denoted by ${\mathbf{\mathcal{W}}}^{(r)}$.

In this paper, when discussing the domain or range of a function, we do not distinguish a linearly ordered set, say, $H^{(r)} = (\{ 1,...,r\} ,\underline { \prec } )$ from its underlying set $\{ 1,...,r\}$. Due to the linearity structure, Yager’s orness definition is very natural and acceptable to measure the extent of bipolar preference within OWA weight vector in numerous applications.

Definition 2.2

[16] The orness of any OWA weight function $w^{(r)}$ is defined as a function $orness:{\mathbf{\mathcal{W}}}^{(r)} \to [0,1]$ such that

$$ orness(w^{(r)} ) = \sum\limits_{i = 1}^{r} {w^{(r)} (i) \cdot \frac{r - i}{{r - 1}}} $$

(2.1)

Dually, the andness of any OWA weight function $w^{(r)}$ is defined as a function $andness:{\mathbf{\mathcal{W}}}^{(r)} \to [0,1]$ by

$$ andness(w^{(r)} ) = \sum\limits_{i = 1}^{r} {w^{(r)} (i) \cdot \frac{i - 1}{{r - 1}}} = 1 - orness(w^{(r)} ) $$

(2.2)

In many applications, the orness/andness can conveniently embody some bipolar decision preference such as optimism/pessimism. In this study, however, we consider the bipolar strictness-indifference decision preference which can also be suitably and well embodied by orness/andness. For example, a 3-dimensional OWA weight vector $w^{(3)} = (0.6,0.3,0.1)$ is with $orness(w^{(3)} ) = 0.75$, representing some strict decision and judgment attitude; a 5-dimensional OWA weight vector $w^{(5)} = (0.2,0.2,0.2,0.2,0.2)$ has $orness(w^{(5)} ) = 0.5$, embodying a neutral attitude to the involved evaluation threshold; and a 4-dimensional OWA weight vector $w^{(4)} = (0.1,0.2,0.3,0.4)$ is with $orness(w^{(4)} ) = 1/3$, indicating a slightly indifferent attitude about the related evaluation standard.

Next, we review the standard definitions related to the space of parameterized OWA weight vectors (of dimension r) and a partial ordering relation on it.

Definition 2.3

[13] For any OWA weight function $w^{(r)} \in {\mathbf{\mathcal{W}}}^{(r)}$, an associated function $\vec{w}^{(r)} :\{ 1,...,r\} \to [0,1]$ such that $\vec{w}^{(r)} (i) = \sum\nolimits_{j = 1}^{i} {w^{(r)} (j)}$ is called the accumulation function of $w^{(r)}$.

Definition 2.4

[13] Given $w_{1}^{(r)} ,w_{2}^{(r)} \in {\mathbf{\mathcal{W}}}^{(r)}$ and $\vec{w}_{1}^{(r)}$, $\vec{w}_{2}^{(r)}$ being their accumulation functions, respectively, if $\vec{w}_{2}^{(r)} \le \vec{w}_{1}^{(r)}$ (i.e., $\vec{w}_{2}^{(r)} (i) \le \vec{w}_{1}^{(r)} (i)$ for all $i \in \{ 1,...,r\}$), then we define $w_{2}^{(r)} \underline { \prec } w_{1}^{(r)}$. When $\vec{w}_{2}^{(r)} \le \vec{w}_{1}^{(r)}$, and there is at least one $k \in \{ 1,...,r - 1\}$ such that $\vec{w}_{2}^{(r)} (i) < \vec{w}_{1}^{(r)} (i)$, we write $w_{2}^{(r)} \prec w_{1}^{(r)}$. With partial ordering $\underline { \prec }$, $({\mathbf{\mathcal{W}}}^{(r)} ,\underline { \prec } )$ is a complete lattice with top element $w_{\max }^{(r)}$ (i.e., $w_{\max }^{(r)} (1) = 1$) and bottom element $w_{\min }^{(r)}$ (i.e., $w_{\min }^{(r)} (r) = 1$).

It is not difficult to find that we have an equivalent expression for the orness of OWA weight function $w^{(r)}$; that is,

$$ orness(w^{(r)} ) = \sum\limits_{i = 1}^{r} {w^{(r)} (i) \cdot \frac{r - i}{{r - 1}}} = \frac{{\left( {\sum\nolimits_{i = 1}^{r} {\vec{w}^{(r)} (i)} } \right) - 1}}{r - 1}. $$

(2.3)

The above formula directly shows that for any two OWA weight functions $w_{1}^{(r)} ,w_{2}^{(r)} \in {\mathbf{\mathcal{W}}}^{(r)}$, if $w_{2}^{(r)} \underline { \prec } w_{1}^{(r)}$, then $orness(w_{2}^{(r)} ) \le orness(w_{1}^{(r)} )$, indicating that the strictness expressed by $w_{2}^{(r)}$ is lower than the one expressed by $w_{1}^{(r)}$.

Next, we review the concept of parameterized family of OWA weight functions (of dimension r).

Definition 2.5

[13] Given $r \in \{ 2,3,...\}$, a function cluster ${\mathcal{W}}^{(r)} = \{ w_{{}}^{(r;\alpha )} \}_{\alpha \in [0,1]} \subseteq {\mathbf{\mathcal{W}}}^{(r)}$ is called a parameterized family of OWA weight functions (of dimension r) if the following two conditions hold:

• 1. for any $\alpha \in [0,1]$, $w_{{}}^{(r;\alpha )} \in {\mathcal{W}}^{(r)}$ and $orness(w_{{}}^{(r;\alpha )} ) = \alpha$;

• 2. for any ${0} \le \alpha_{{1}} < \alpha_{2} \le 1$, $w_{{}}^{{(r;\alpha_{1} )}} \prec w_{{}}^{{(r;\alpha_{2} )}}$.

With the accumulation functions $\vec{w}_{{}}^{(r;\alpha )}$, we observe that $\vec{w}_{{}}^{(r;\alpha )} (r) = 1$ for any $\alpha \in [0,1]$; and for any $i \in \{ 1,...,r - 1\}$, $\vec{w}_{{}}^{(r;0)} (i) = 0$ and $\vec{w}_{{}}^{(r;1)} (i) = 1$.

We recall two-parameterized family of OWA weight functions. The binomial family ${\mathcal{W}}^{(r;B)} = \{ w_{{}}^{(r;\alpha ;B)} \}_{\alpha \in [0,1]}$ [17] satisfies

$$ w_{{}}^{(r;\alpha ;B)} (i) = \left( \begin{gathered} r - 1 \hfill \\ i - 1 \hfill \\ \end{gathered} \right)(1 - \alpha )^{i - 1} \alpha^{r - i} . $$

(2.4)

The Hurwicz family, ${\mathcal{W}}^{(r;H)} = \{ w_{{}}^{(r;\alpha :H)} \}_{\alpha \in [0,1]}$ such that $w_{{}}^{(r;\alpha :H)} (1) = \alpha$ and $w_{{}}^{(r;\alpha :H)} (r) = 1 - \alpha$. Note that when $r = 2$, there only exists the sole family ${\mathcal{W}}^{(2)} = \{ w^{(2;\alpha )} \}_{\alpha \in [0,1]} = \{ (\alpha ,1 - \alpha )\}_{\alpha \in [0,1]} = {\mathbf{\mathcal{W}}}^{(2)}$. In the rest of this paper, we will only adopt the Binomial family for use.

Since in the evaluation environment of this study it is needed to handle heterogeneous linguistic input information which will be expressed as several different OWA weight functions with varying dimensions, then we next extend the concept of set of OWA weight functions ${\mathbf{\mathcal{W}}}^{(r)}$ in Definition 2.1 and propose the extended set of OWA weight functions.

Definition 2.6

An extended space of OWA weight functions ${\mathbf{\mathcal{W}}}^{(r)}$ with varying dimensions $r \in \{ 2,3,...\}$ is denoted by ${\mathbf{\mathbb{W}}} = \bigcup\nolimits_{r = 2}^{\infty } {{\mathbf{\mathcal{W}}}^{(r)} }$.

With this definition, we will have the following extended inputs information which accommodates heterogeneous linguistic input information and can be also defined by a function.

Definition 2.7

The extended inputs information (for evaluation in this study) is defined as a function $x:\{ 1,...,n\} \to {\mathbf{\mathbb{W}}}$.

For example, an extended inputs information could be a function $x:N^{(3)} \to {\mathbf{\mathbb{W}}}$ such that $x(1) = (0.6,0.4)$, $x(2) = (0.4,0.2,0.3,0.1)$ and $x(3) = (0.3,0.5,0.2)$.

For making the discussion and formulation better and clear, we distinguish and strictly present the following two definitions about linguistic evaluation and distributional linguistic evaluation.

Definition 2.8

• 1. When referring to a sole value on a linguistic term set $H^{(r)}$, a 0–1 valued function $G:H^{(r)} \to \{ 0,1\}$, satisfying $\sum\nolimits_{i = 1}^{r} {G(i)} = 1$, is called a linguistic evaluation.

• 2. When a normalized distribution is obtained on a linguistic term set $H^{(r)}$, a normalized distribution $p:H^{(r)} \to [0,1]$ (satisfying $\sum\nolimits_{i = 1}^{r} {p(i)} = 1$) is called a distributional linguistic evaluation.

Remark

In some different decisional scenarios, with the same function $p:H^{(r)} \to [0,1]$ we also call it an OWA weight function without any confusion. Besides, $p:H^{(2)} \to [0,1]$ can be equivalently regarded as a real value $a \in [0,1]$ if $a = p(1)$.

Comprehensive Evaluation Using Linguistic Evaluation Thresholds for Heterogeneous Inputs

As we have discussed in the preceding section, a 3-scale linguistic term set $H^{(3)}$ = ({1 “optimal”, 2 “average”, 3 “substandard”},$\underline { \prec }$) is taken as the evaluation scale for judging the real evaluation result $E(A) \in [0,1]$ of the evaluation object A. In there an evaluation thresholds $(a,b) \in [0,1]^{2}$ is applied to partition [0,1] and to decide the linguistic evaluation of A according to which area $E(A)$ falls into.

When the single evaluation result or individual evaluation input is a function value of the extended inputs information $x:N^{(n)} \to {\mathbf{\mathbb{W}}}$ (i.e., $x(i) = (v_{1} ,...,v_{{r_{i} }} )$ is the individual distributional linguistic evaluation for A with respect to criterion $C_{i}$), then we need to partition ${\mathbf{\mathcal{W}}}^{{(r_{i} )}}$ into three pairwise disjoint subsets and observe which subset $x(i)$ falls into. To fulfill this aim, we firstly still need a real-valued evaluation thresholds $(a,b) \in [0,1]^{2}$ (with $a \le b$) to derive an OWA weight-valued evaluation thresholds $(w_{{}}^{{(r_{i} ;a;B)}} ,w_{{}}^{{(r_{i} ;b;B)}} ) \in \left( {{\mathcal{W}}^{{(r_{i} ;B)}} } \right)^{2}$ with $w_{{}}^{{(r_{i} ;a;B)}} \underline { \prec } w_{{}}^{{(r_{i} ;b;B)}}$. Then, we can categorize $x(i)$ into the three partitioned subsets by the following rules, similar to what have been put in Step 8 of the basic evaluation method using threshold values near the beginning of Sect. 2:

• If $w_{{}}^{{(r_{i} ;b;B)}} \underline { \prec } x(i)$, then the linguistic judgment of A with respect to $C_{i}$ is 1 “optimal”;

• if $x(i)\underline { \prec } w_{{}}^{{(r_{i} ;a;B)}}$, then the linguistic judgment is 3 “substandard”;

• else, the linguistic judgment is 2 “average”.

A complete set of procedures are organized and proposed in what follows.

Remark 3.1

The OWA weight-valued evaluation thresholds used in the above procedures all correspond to a 3-scale linguistic term $H^{(3)}$ = ({1 “substandard”, 2 “average”, 3 “optimal”},$\underline { \prec }$), which can be extended to some terms with more scales if wanted. The 3-scale linguistic term set we used is practical and workable in application because it is has a good affinity and close relation to intuitionistic fuzzy sets [23] which is commonly applied in numerous applications [24]. In addition, to adopt the linguistic term set with relatively lower dimension could present a clearer illustration for practitioners to understand the proposed evaluation problem.

An application in Reservoir Operation Quality and Effect Evaluation

This section provides a numerical case of the proposed evaluation model in the evaluation of reservoir operation and effect. Reservoir operation is related to the water resource management and evaluation [25] and is important in many aspects of social development and environment conservation.

We will evaluate a certain reservoir A in the southern area of Nanjing about its operation quality and effect and will adopt a set of four criteria after consulting with some experts working in that reservoir. The evaluation result is useful for further planning and possible improvement or adjustment of that reservoir.

We next elaborate the evaluation procedures proposed in the preceding section. All the initial data and linguistic information are provided by some experts working in that reservoir or studying in water recourse management and planning.

We make some final comments and discussions. With the heterogeneous linguistic input information $x:\{ 1,...,n\} \to {\mathbf{\mathbb{W}}}$ and a set of linguistic term set $\{ H_{i}^{{(r_{i} )}} \}_{i = 1}^{n}$ with different numbers of scale, we can indeed devise a set of valuating rules to directly transform x into a real function just like a piece of real valued input information. For example, $x(3) = (0.7,0.2,0.1,0)$ can be transformed into a real value by performing a weighted average with a reasonably designed score vector, say, $q = (1,0.7,0.3,0)$, and obtain a real value $y(3) = (0.7)(1) + (0.2)(0.7) + (0.1)(0.3) + (0)(0) = 0.87$. Then, we can perform some commonly known evaluation methods using the obtained real function y. However, the score vectors for transforming are not always easy to design. As for the evaluation method using linguistic evaluation thresholds, apart from its clear reasonability and feasibility, another advantage of it lies in that one can easily and flexibly add some more restrictions to the rules deciding the linguistic evaluation results. For example, at some certain decision situations, one may add some more conditions for achieving a 1 “optimal” evaluation; that is, for example, we can add one more condition “$\left( {x(r_{i} )} \right)(1) \ge 0.8$” to the original condition “$w_{i}^{{(r_{i} ;b;B)}} \underline { \prec } x(r_{i} )$”. Clearly, the proposed model is more adaptive and flexible than the commonly used evaluation methods based on merging real values and then judging.

Conclusions

Thresholds-based evaluation is commonly seen, workable and effective in many practical evaluation problems including the one involving in multiple criteria. When the individual inputs corresponding to each evaluation criteria are real numbered, we can be relatively easy to devise some reasonable evaluation schemes and procedures to perform the desired evaluation.

When the individual inputs are provided by heterogeneous linguistic information with distributional forms, the extended space of OWA weight functions was defined to help strictly devise some partitioning methods for those distributional inputs. The proposed extended space has good adaptivity since it can accommodate different dimensions. In addition, we defined a linguistic evaluation to be a function on linearly ordered linguistic term set, while we also defined a distributional linguistic evaluation to be an OWA weight function. Besides, the distributional information derived can be obtained from statistics and thus may have more objectivity than other linguistic decision making methods.

We adopted a 3-scale linguistic term set mainly for practical purpose and illustrative convenience. In actual, the linguistic term set can be extended into higher dimension according to real needs. By a two-layer evaluation procedures, a linguistic comprehensive evaluation is well built and applied to reservoir evaluation.