What you think and what I think: Studying intersubjectivity in knowledge artifacts evaluation

Abstract

Miscalibration, the failure to accurately evaluate one’s own work relative to others' evaluation, is a common concern in social systems of knowledge creation where participants act as both creators and evaluators. Theories of social norming hold that individual’s self-evaluation miscalibration diminishes over multiple iterations of creator-evaluator interactions and shared understanding emerges. This paper explores intersubjectivity and the longitudinal dynamics of miscalibration between creators' and evaluators' assessments in IT-enabled social knowledge creation and refinement systems. Using Latent Growth Modeling, we investigated dynamics of creator’s assessments of their own knowledge artifacts compared to peer evaluators' to determine whether miscalibration attenuates over multiple interactions. Contrary to theory, we found that creator’s self-assessment miscalibration does not attenuate over repeated interactions. Moreover, depending on the degree of difference, we found self-assessment miscalibration to amplify over time with knowledge artifact creators' diverging farther from their peers' collective opinion. Deeper analysis found no significant evidence of the influence of bias and controversy on miscalibration. Therefore, relying on social norming to correct miscalibration in knowledge creation environments (e.g., social media interactions) may not function as expected.

Appendices

Appendix 1: Measurement Operationalization (Single-Group Ranking Model)

Suppose, each peer group consists of N subjects that are indexed i = {1, 2, …, N}. Each subject i rank-orders other (N-1) subjects' Artifacts so that the “best” is ranked 1 and the “worst” is ranked (N-1), that is, here only subject i’s rank-ordering of his peers' Artifacts is considered. The matrix of mutual evaluations of Artifacts produced by the group is

$$ {\boldsymbol{A}}_{N\times N}={\left[{a}_{ij}\right]}_{N\times N}=\left[\begin{array}{ccc}\begin{array}{cc}N& {a}_{12}\\ {}{a}_{21}& N\end{array}& \cdots & \begin{array}{c}{a}_{1N}\\ {}{a}_{2N}\end{array}\\ {}\vdots & \ddots & \vdots \\ {}{a}_{N1}\kern0.5em {a}_{N2}& \cdots & N\end{array}\right] $$

where ranks given are in rows, ranks received are in columns, a _ij denotes a rank given by a subject i to a subject j for the Artifact (or, symmetrically, received by a subject j from a subject i). Note that a _i  = [a _i1  a _i2 … a _ij … a _iN ] is a row vector of ranks given by a subject i to all his peers such that

$$ \left\{\begin{array}{c}{a}_{ij}=N\ if\ i=j\ \\ {}{a}_{ij}\in \left\{1,2,\dots, N-1\right\}\ if\ i\ne j\ \\ {}{a}_{i1}\ne {a}_{i2}\ne \dots \ne {a}_{ij}\ne \dots \ne {a}_{iN}\\ {}{a}_{ij}=N\ if\ {E}_j=0\end{array}\right. $$

where E _j is the indicator function such that

$$ {E}_j=\left\{\begin{array}{c}1\ if\ the\ Artifact\ was\ submitted\ by\ a\ subject\ j\ \\ {}0\ if\ the\ Artifact\ was\ not\ submitted\ by\ a\ subject\ j\end{array}\right. $$

Note that the row vector e = [E ₁  E ₂ … E _N ] is the vector of the indicator function’s values. These conditions constitute the data integrity constraints. The first condition means that a subject i’s assessment of his own Submission is not included in this matrix. The second condition means that each peer’s Artifact, without exceptions, needs to be rank-ordered. The third condition means that rank-ordering is enforced, that is, no two peers' Artifacts may have the same rank. Note that relaxing this assumption allows each Artifact to be rated rather than ranked. The forth condition means that a missing Artifact is given N points.

Suppose C > 1 is the maximum possible attainment score for an Artifact; i.e., the attainment score of C is given to an Artifact that received the rank of 1, and the attainment score of 1 is given to an Artifact that received the rank of (N-1). A failure to submit an Artifact results in the attainment score of 0. Then the following transforms the rank a _ij given by a subject i to subject j into the attainment score c _ji received by subject j from subject i:

$$ {c}_{ji}=\left\{\begin{array}{c}1+\left(\sum_{h=1}^N{E}_h-1-{a}_{ij}\right)\frac{D-1}{\sum_{h=1}^N{E}_h-2}\ if\ {a}_{ij}\ne N\\ {}0\ if\ {a}_{ij}=N,\end{array}\right. $$

or

$$ {c}_{ji}=\left\{\begin{array}{c}{a}_{ij}\frac{\left(1-C\right)}{\sum_{h=1}^N{E}_h-2}+\frac{C\left({\sum}_{h=1}^N{E}_h-1\right)-1}{\sum_{h=1}^N{E}_h-2}\ if\ {a}_{ij}\ne N\\ {}0\ if\ {a}_{ij}=N\end{array}\right. $$

For example, suppose N = 6 subjects and C = 5 points. Then, the transformation rule is:

Rank a ij	Attainment score c ij
1	5
2	4
3	3
4	2
5	1
Not submitted (6)	0

The matrix of the individual Artifact attainment scores for the entire group is (scores received are in rows, scores given are in columns)

$$ {\boldsymbol{C}}_{N\times N}={\boldsymbol{A}}^{'}\frac{\left(1-C\right)}{1{\boldsymbol{e}}^{'}-2}+{1}^{'}1\ \frac{C\left(1{\boldsymbol{e}}^{'}-1\right)-1}{1{\boldsymbol{e}}^{'}-2} $$

where scores received are in rows, scores given are in columns,

1 _1×N = [1 1 … 1] is the row vector of ones,

c _ji  = 0 for all a _ji  = N, and

$$ 1{\boldsymbol{e}}^{\boldsymbol{\hbox{'}}}=\sum_{j=1}^N{E}_j\le N. $$

A subject i’s attainment score for the Artifact is the average attainment score received from all his peers, who submitted their evaluations, ideally (N-1). Hence, the column vector of attainment scores for Artifacts is

$$ \overset{-}{\boldsymbol{c}}=\frac{\boldsymbol{C}\ {1}^{\hbox{'}}}{1{\boldsymbol{f}}^{\hbox{'}}-1} $$

where the row vector f = [F ₁  F ₂ … F _N ] is the vector of values of the indicator function F _j such that

$$ {F}_j=\left\{\begin{array}{c}1\ if\ the\ Artifact\ Evaluation\ was\ submitted\ by\ subject\ j\ \\ {}0\ if\ the\ Artifact\ Evaluation\ was\ not\ submitted\ by\ subject\ j,\end{array}\right. $$

$$ 1{\boldsymbol{f}}^{\boldsymbol{\hbox{'}}}=\sum_{j=1}^N{F}_j\le N $$

Hence, the attainment score for the Artifact of subject i is

$$ {\overset{-}{c}}_i=\frac{{\boldsymbol{c}}_i\ {1}^{\hbox{'}}}{1{\boldsymbol{f}}^{\mathbf{\hbox{'}}}-1} $$

where ${{\boldsymbol{c}}_i}_{1\times N}$ is the row vector of Artifact attainment scores received by a subject i.

Subjects are also required to self-evaluate, that is, rank-order their own Artifacts among those of other peers. However, their self-ranking is not included in the calculation of attainment of their own Artifact. The difference between attainment measures derived from self-assessment and peer assessment results is computed as follows. Suppose α _i is a rank given by a subject i to his own Artifact, such that α _i coincides with one of the values a _i . Then, self-evaluation score for Artifact attainment is defined as

$$ {\varepsilon}_i=\left\{\begin{array}{c}\left(1+\left({N-\alpha}_i\right)\frac{C-1}{N-1}\right)\ if\ {F}_i=1\\ {}\varnothing\ if\ {F}_i=0,\end{array}\right. $$

and miscalibration, i.e., the difference between the attainment score produced by self-assessment and the attainment score produced by peer assessment) for the Artifact of a student i is defined as

$$ {\varDelta}_i=\left\{\begin{array}{c}\frac{{\overset{-}{c}}_i-{\varepsilon}_i}{C-1}\ if\ {F}_i=1\\ {}\varnothing\ if\ {F}_i=0.\end{array}\right. $$

The assessor error (ER) measures a given subject’s divergence from assessments of each peer’s Artifact by the rest of the peer group. Subject i’s assessor error is defined as

$$ {\delta}_i=\left\{\begin{array}{c}\sum_{j=1}^N\left|{\overset{-}{c}}_j-{c}_{ji}\right|\ \forall\ i\ \ne j\ if\ {F}_i=1\\ {}\varnothing\ if\ {F}_i=0\end{array}\right. $$

where |. | denotes the absolute value operator, and ø denotes an undefined (missing) value (assessor error cannot be calculated if assessment was not submitted). The column vector of assessor error measures is

$$ \boldsymbol{\delta} =\left|{\left(\overset{-}{\boldsymbol{c}}1\right)}^{\hbox{'}}-\boldsymbol{C}\hbox{'}\right|\ {1}^{\hbox{'}} $$

where |. | denotes the matrix of absolute values of the element-wise differences of the two square matrices (not the determinant of the matrix). In this vector, all elements for which F _i is zero are undefined (missing values).

The assessee error (EE) is a measure of divergence among peers in assessing a given subject’s Artifact. Subject i’s assesse error is defined as

$$ {\gamma}_i=\left\{\begin{array}{c}\sum_{j=1}^N\left|{\overset{-}{c}}_i-{c}_{ij}\right|\ \forall\ i\ \ne j\ if\ {E}_i=1\\ {}\varnothing\ if\ {E}_i=0\end{array}\right. $$

The column vector of assesse error measures is

$$ \boldsymbol{\gamma} =\left(\left|\overset{-}{\boldsymbol{c}}1-\boldsymbol{C}\right|\ {}^{\circ}\ \boldsymbol{Q}\right)\ {1}^{\hbox{'}} $$

where Q _N×N is a square matrix with zeros on the diagonal and ones off diagonal, the operator “°” denotes the Hadamard product of matrices (entry-wise product operator). In this vector, all elements for which E _i is zero are undefined (missing values).

The average group assessor error (AGER) is defined as

$$ \overset{-}{\delta }=\frac{1\boldsymbol{\delta}\ }{1{\boldsymbol{f}}^{\boldsymbol{\hbox{'}}}}=\frac{\sum_{i=1}^N{\delta}_i}{\sum_{i=1}^N{F}_i} $$

The average group assessee error (AGEE) is defined as

$$ \overset{-}{\gamma }=\frac{1\boldsymbol{\gamma}\ }{1{\boldsymbol{e}}^{\boldsymbol{\hbox{'}}}}=\frac{\sum_{i=1}^N{\gamma}_i}{\sum_{i=1}^N{E}_i} $$

It can be shown that corresponding AGER and AGEE are equal.

The intra-group inter-observer reliability (IGIOR) for any given group is defined as

$$ y=\frac{{\overset{-}{\delta}}_{div}-\overset{-}{\delta }}{{\overset{-}{\delta}}_{div}-{\overset{-}{\delta}}_{con}} $$

where $\overset{-}{\delta }$ is the AGER of the given group,

${\overset{-}{\delta}}_{con}$ is the AGER of the group with the perfect convergence among peers’ evaluations of each other’s Artifacts on the ordinal-scale (relative ranks),

${\overset{-}{\delta}}_{div}$ is the AGER of the group with the perfect divergence among peers’ evaluations of each other’s Artifacts on the ordinal-scale (relative ranks).

The IGIOR can be interpreted as how far the given peer group as a whole is from the perfect convergence on ranking each other’s Artifact. For a group with the perfect convergence, the IGIOR is equal 1; for a group with the perfect divergence, the IGIOR is equal 0. Note that since it can be shown that corresponding AGER and AGEE are equal, it does not matter whether either AGER or AGEE are used to compute the IGIOR.

Bias (normalized ER) is the ER adjusted for the chosen values of C so that it ranges between 0 and 1. Bias can be interpreted as a measure of a given subject’s divergence from the rest of the peer group in assessing each peer’s Artifact irrespective of the overall rank of the subject’s Artifact and the maximum possible attainment score.

A given subject i’s bias is defined as

$$ {\widehat{\delta}}_i\left({\delta}_i,{r}_i\ \right)=\frac{\delta_i-y\ {\delta}_{con}\left({r}_i\left({\overset{-}{c}}_i\right)\right)}{\delta_{div}} $$

where ${r}_i$is the rank of the subject i’s Artifact among the rest of the Artifacts of the peer group based on the Artifact’s attainment score (in other words, ${r}_i$ corresponding to the larges value in the vector $\overset{-}{\boldsymbol{c}}$ is equal to 1 and ${r}_i$ corresponding to the smallest value in the vector $\overset{-}{\boldsymbol{c}}$ is equal to N);

${\delta}_{con}\left({r}_i\right)$ is the ER corresponding to the rank ${r}_i$ in the peer groups with the perfect convergence (IGIOR y = 1);

${\delta}_{div}$ is the ER in the peer groups with the perfect divergence (IGIOR y = 0). In a peer groups with the perfect divergence, all peers have the same ER because no one is better in assessing others than the rest of the group. Computations of ${\delta}_{con}\left({r}_i\right)$ and ${\delta}_{div}$ are explained in the appendix 2.

Controversy (normalized EE) is the EE adjusted for the chosen values of C so that it ranges between 0 and 1. Controversy can be interpreted as a measure of divergence among peers in assessment of a given subject’s Artifact irrespective of the overall ranks their Artifacts and the maximum possible attainment score for the Artifact.

Controversy of an Artifact produced by a subject i is defined as

$$ {\widehat{\gamma}}_i\left({\gamma}_i,{r}_i\right)=\frac{\gamma_i-y\kern1em {\gamma}_{con}\left({r}_i\left({\overline{c}}_i\right)\right)}{\gamma_{div}}. $$

Bias and controversy are recorded in vectors $\widehat{\boldsymbol{\delta}}$ and $\widehat{\boldsymbol{\gamma}}$ respectively. Normalization is necessary because while ER and EE are the same for all subjects in the peer group with the perfect divergence (because no one is better in assessing Artifacts than the rest of the group), in the peer group with the perfect convergence ER and EE have non-zero values and are non-linearly dependent on the rank r _i of the subject i’s Artifact among the Artifacts of the peer group based on the attainment score. In other words, in a peer group with non-equal attainment scores (that is when at least some convergence exists among peer on the quality of each submission and submissions can be ranked based on the score), each rank position is characterized by a systematic non-zero ER and EE just because of its place in the relative ranking of peers' Artifacts. This is due to the fact that subjects’ own self-evaluations are not included in the computations of the attainment scores. A more detailed explanation of this phenomenon is given below.

This single-groups ranking model can be easily extended to multiple groups.

Appendix 2: Correcting Nonlinearity in Ranking (Computing AGER and AGEE for Peer Groups with Perfect Convergence and Perfect Divergence)

To determine IGIOR, bias and controversy, AGER and AGEE for the extreme special cases of the perfect convergence and the perfect divergence of a peer group need to be computed. In addition, ER for each relative rank position in a peer group needs to be computed for the case of perfect convergence.

First, let us consider the case of the perfect intra-group inter-observer convergence, that is, the summative assessment result for which IGIOR is equal 1. Suppose the row vector s _{1 × N}  = [1, 2, …, N] is the vector of latent ranks of potential attainment (goodness) of a given set of Artifacts. That is, we assume that each Artifact is of such goodness and each subject has such evaluation skills that when asked to rank-order these Artifacts, the subjects come to the perfect convergence on ranking of each Artifact (we also assume that all subjects submit their Artifacts). Under these assumptions, the latent ranks should be equal to the ranks generated by the system, that is s _i  = r _i for all i. However, since each Artifact’s attainment is computed using ranks received from peers (ranging in {1, 2, …, N-1}) and excluding subject’s self-ranking of his own Artifact, despite the perfect convergence each subject will make an assessor error (ER) of a various degree. For example, in a peer group of six subjects, subject 1 will not be able to give his own Artifact the rank of 1 but would have to give it to the Artifact of subject 2. Similarly, subjects 2, 3, 4, and 5 will have to give the rank of 5 to the Artifact of subject 6. Consequently, each Artifact will also bear an assessee error (EE) of varying degree.

The matrix A _con of ranks given in a peer group with the perfect convergence is obtained from the vector s by the following transformation

$$ {\boldsymbol{A}}_{con}=N\ \boldsymbol{I}+\left({1}^{\hbox{'}}\boldsymbol{s}\ {\boldsymbol{H}}_1\right){}^{\circ}\ {\boldsymbol{T}}_1+\left({1}^{\hbox{'}}\boldsymbol{s}\ {\boldsymbol{H}}_1{\boldsymbol{H}}_2\right){}^{\circ}\ {\boldsymbol{T}}_2 $$

where I _N×N is the identity matrix, 1 _1×N is a vector of ones,

H _{1 N×N} is a square shift matrix with all elements equal zero except for elements equal one just above the main diagonal,

H _{2 N×N} is a square shift matrix with all elements equal zero except for elements equal one just below the main diagonal,

T _{1 N×N} is a square matrix with all elements in the upper triangle above the main diagonal equal ones and all other equal zero,

T _{2 N×N} is a square matrix with all elements in the lower triangle below the main diagonal equal ones and all other equal zero, the operator “°” denotes the Hadamard product of matrices (entry-wise product operator).

For the special case of N = 6, the matrix A _con looks as follows:

6	1	2	3	4	5
1	6	2	3	4	5
1	2	6	3	4	5
1	2	3	6	4	5
1	2	3	4	6	5
1	2	3	4	5	6

Using the rule for transforming ranks into scores the matrix of scores C _con is obtained as

$$ {\boldsymbol{C}}_{con}={{\boldsymbol{A}}_{con}}^{'}\frac{\left(1-C\right)}{N-2}+{1}^{'}1\ \frac{C\left(N-1\right)-1}{N-2} $$

such that

$$ {c}_{ji}=\left\{\begin{array}{c}1+\left(N-{a}_{ij}-1\right)\frac{C-1}{N-2}={a}_{ij}\frac{\left(1-C\right)}{N-2}+\frac{C\left(N-1\right)-1}{N-2}\ if\ {a}_{ij}\ne N\\ {}0\ if\ {a}_{ij}=N\end{array}\right. $$

For the case of N = 6 and C = 5, the matrix C _con looks as follows:

0.00	5.00	5.00	5.00	5.00	5.00
5.00	0.00	4.00	4.00	4.00	4.00
4.00	4.00	0.00	3.00	3.00	3.00
3.00	3.00	3.00	0.00	2.00	2.00
2.00	2.00	2.00	2.00	0.00	1.00
1.00	1.00	1.00	1.00	1.00	0.00

The column vector of attainment scores for the peer group in the perfect convergence is

$$ {\overset{-}{\boldsymbol{c}}}_{con}={\frac{{\boldsymbol{C}}_{con}}{\ 1}}^{'}1{\boldsymbol{f}}^{'} $$

The column vector of ER is

$$ {\boldsymbol{\delta}}_{con}=\left|{\left({\overset{-}{\boldsymbol{c}}}_{con}1\right)}^{'}-{\boldsymbol{C}}_{con}'\right|\ {1}^{'} $$

where |. | denotes the matrix of absolute values of the element-wise differences of the two square matrices (not the determinant of the matrix).

The column vector of EE is

$$ {\boldsymbol{\gamma}}_A=\left(\left|{\overset{-}{\boldsymbol{c}}}_{con}1-{\boldsymbol{C}}_{con}\right|{}^{\circ }\ \boldsymbol{Q}\right)\ {1}^{'} $$

where Q _N×N is a square matrix with zeros on the diagonal and ones off diagonal, the operator “°” denotes the Hadamard product of matrices (entry-wise product operator).

The following table summarizes the attainment, assessor error (ER) and assessee errors (EE) scores for the case of the peer group with the perfect convergence where N = 6 and C = 5:

s = r	${\overset{-}{c}}_{con}$	${\delta}_{con}(r)$	${y}_{con}(r)$
1	5.00	2.00	0.00
2	4.20	1.20	1.60
3	3.40	0.80	2.40
4	2.60	0.80	2.40
5	1.80	1.20	1.60
6	1.00	2.00	0.00

The following diagram graphically represents attainment, ER and EE scores for the extreme special case of the peer group with the perfect convergence where N = 6 and C = 5:

Fig. 13

Non-linear Error Behavior in the Perfect Convergence Group (N = 6, C = 5)

Thus, we obtained ER and EE scores for the extreme special case of the peer group with the perfect convergence.

The AGER for the peer groups with the perfect convergence is

$$ {\overset{-}{\delta}}_{con}=\frac{1{\boldsymbol{\delta}}_{con}\ }{1{\boldsymbol{f}}^{'}}=\frac{\sum_{i=1}^N{\delta}_{con\ i}}{\sum_{i=1}^N{F}_i}. $$

The AGEE for the peer groups with the perfect convergence is

$$ {\overset{-}{\gamma}}_{con}=\frac{1{\boldsymbol{\gamma}}_{con}\ }{1{\boldsymbol{e}}^{'}}=\frac{\sum_{i=1}^N{\gamma}_{con\ i}}{\sum_{i=1}^N{E}_i}. $$

For the peer group with the perfect convergence with N = 6 and C = 5, the AGER ${\overset{-}{\delta}}_{con}$and the AGEE ${\overset{-}{\gamma}}_{con}$ are both equal 4/3.

Thus, we obtained values of AGER and AGEE for each relative rank for the extreme case of the perfect convergence.

Now, let us consider the case of the perfect intra-group inter-observer divergence, that is, the result of mutual summative peer assessment, for which IGIOR is equal zero. In this case, the row vector s _1×N  = [1, 2, …, N] does not reflect latent ranks of quality of a given set of Artifacts. That is, we assume that each Artifact is of such goodness, and each subject has such evaluation skills that when asked to rank-order these Artifacts the subjects are not be able to converge on the goodness of Artifacts and their ranking, resulting in the perfect divergence on ranking of each Artifact (we also again assume that all subjects submit the deliverable). In other words, the latent goodness of all Artifacts and evaluation skills of all subjects are assumed to be absolutely equivalent/homogenous. Under these assumptions, each subject’s Artifact receives from his peers the entire range of possible ranks, and no one subject is better in assessing his peers than the rest of the group. The matrix A _div of ranks given in a peer group with the perfect divergence, therefore, is a Latin square – an N × N matrix filled with integers from {1, 2, …, N}, each occurring exactly once in each row and exactly once in each column. One of the ways such matrix can be constructed is by the Cyclic Method (Bailey, 2008): Place s in reverse order (or $\left(N+1\right)1-\boldsymbol{s}$) in the top row of A _div ; in the second row, shift all the integers to the right one place, moving the last symbol to the front; continue in this fashion, shifting each row one place to the right of the previous row.

For the special case of N = 6, the matrix A _div looks as follows:

6	5	4	3	2	1
1	6	5	4	3	2
2	1	6	5	4	3
3	2	1	6	5	4
4	3	2	1	6	5
5	4	3	2	1	6

Note that for the purpose of obtaining ER and EE scores for the case of the perfect divergence the method of obtaining A _div matrix does not matter as long as it is a Latin square with the diagonal elements equal N.

Similarly to the case of the perfect convergence, using the rule for transforming ranks into scores the matrix of attainment scores C _div is obtained as

$$ {\boldsymbol{C}}_{div}={{\boldsymbol{A}}_{div}}^{'}\frac{\left(1-C\right)}{N-2}+{1}^{'}1\ \frac{C\left(N-1\right)-1}{N-2} $$

such that

$$ {c}_{ji}=\left\{\begin{array}{c}1+\left(N-{a}_{ij}-1\right)\frac{C-1}{N-2}={a}_{ij}\frac{\left(1-C\right)}{N-2}+\frac{C\left(N-1\right)-1}{N-2}\ if\ {a}_{ij}\ne N\\ {}0\ if\ {a}_{ij}=N\end{array}\right. $$

For the case of N = 6 and C = 5, the matrix C _div looks as follows:

0.00	5.00	4.00	3.00	2.00	1.00
1.00	0.00	5.00	4.00	3.00	2.00
2.00	1.00	0.00	5.00	4.00	3.00
3.00	2.00	1.00	0.00	5.00	4.00
4.00	3.00	2.00	1.00	0.00	5.00
5.00	4.00	3.00	2.00	1.00	0.00

The column vector of attainment scores for the group in the perfect convergence is

$$ {\overset{-}{\boldsymbol{c}}}_{div}={\frac{{\boldsymbol{C}}_{div}}{\ 1}}^{'}1{\boldsymbol{f}}^{'} $$

The column vector of assessor errors is

$$ {\boldsymbol{\delta}}_{div}=\left|{\left({\overset{-}{\boldsymbol{c}}}_{div}1\right)}^{'}-{\boldsymbol{C}}_{div}'\right|\ {1}^{'} $$

The column vector of assessee errors is

$$ {\boldsymbol{\gamma}}_{div}=\left(\left|{\overset{-}{\boldsymbol{c}}}_{div}1-{\boldsymbol{C}}_{div}\right|{}^{\circ }\ \boldsymbol{Q}\right)\ {1}^{'} $$

The following table summarizes attainment, bias and controversy scores for the case of the peer group with the perfect divergence where N = 6 and C = 5:

s = r	${\overset{-}{c}}_{div}$	${\delta}_{div}(r)$	${\gamma}_{div}(r)$
1	3.00	6.00	6.00
2	3.00	6.00	6.00
3	3.00	6.00	6.00
4	3.00	6.00	6.00
5	3.00	6.00	6.00
6	3.00	6.00	6.00

Thus, we obtained values of ER and EE for the extreme special case of the peer group with the perfect divergence. Note that all subjects' submissions in such group are characterized by equal attainment, ER and EE scores.

The AGER for the peer group with the perfect divergence is

$$ {\overset{-}{\delta}}_{div}=\frac{1{\boldsymbol{\delta}}_{div}\ }{1{\boldsymbol{f}}^{'}}=\frac{\sum_{i=1}^{N-1}{\delta}_{div\ i}}{\sum_{i=1}^{N-1}{F}_i}. $$

The AGEE for the peer group with the perfect divergence is

$$ {\overset{-}{\gamma}}_{div}=\frac{1{\boldsymbol{\gamma}}_{div}\ }{1{\boldsymbol{e}}^{\mathbf{\hbox{'}}}}=\frac{\sum_{i=1}^{N-1}{\gamma}_{div\ i}}{\sum_{i=1}^{N-1}{E}_i}. $$

For the peer group with the perfect divergence where N = 6 and C = 5, AGER ${\overset{-}{\delta}}_{div}$and AGEE ${\overset{-}{\gamma}}_{div}$ are both equal 6.

Thus, we obtained values of AGER and AGEE for the extreme special case of the perfect divergence. The IGIOR for the case of the perfect convergence ${y}_{con}=1$ and for the case of the perfect divergence ${y}_{div}=0.$