Assessing the Effectiveness of Reporting Systems: Why and How

Abstract

This chapter proposes and describes a methodology to assess the effectiveness of the reporting systems for responsible tourism and CSR available in Europe, the scope being to give tourism businesses a set of elements useful to choose the program that best responds to their characteristics and requirements. Through the application of a mathematical and multi-criteria decision model, the Analytic Hierarchy Process (AHP), the main characteristics, strengths and weaknesses of every program have been evaluated and compared, according to different criteria. Better than a qualitative analysis, AHP analyses the performance of every alternative with respect to each parameter and finally calculates a value that expresses the overall quality and effectiveness of each system in assessing business responsibility. In particular, the main criteria that have been selected are as follows: the degree of attention paid by every program to all triple bottom line dimensions; the opportunity for small and medium-sized enterprises to apply the system; the integration between the “certification approach” and the “responsible tourism approach”; the transparency given by the type of auditing. The method selects the best reporting system to apply, according to the importance given to each criterion taken into account.

Appendix: Some Details About the AHP Procedure

This appendix proposes a further explanation of the AHP model, in particular of the calculation of weights, starting from the pair comparison technique, and of the consistency verification.

Determining the weights of the alternatives signifies assigning to them a numerical value that expresses their suitability and importance compared to the other options for each of the criteria. AHP requires to assign priorities starting from the technique of pair comparison, in other words, by comparing two by two all the alternatives, with respect to each single sub-criterion, and questioning whether one is preferable to another and to what extent.

The operation must be entrusted to experts familiar with the subject, who will be asked to express their personal preferences according to a scale developed by Saaty:

• 1 If alternative i and alternative j are equal;

• 3 If alternative i is moderately preferable to alternative j;

• 5 If alternative i is strongly preferable to alternative j;

• 7 If alternative i is more strongly preferable to alternative j;

• 9 If alternative i is extremely preferable to alternative j;

• 2,4,6,8 As intermediate values (or in the case of compromise).

The result of the comparison is the coefficient of dominance a _ij , an estimate of the dominance of the first element i) compared to the second (j). In particular, making a paired comparison of n elements results in n ² coefficients, of which only n(n−1)/2 must be directly determined by the expert who is carrying out the evaluation, in that:

$$ {a}_{ii}=1, $$

(7.1)

and

$$ {a}_{ji}=1/{a}_{ij} $$

(7.2)

for each of the values of i and j.

The coefficients of dominance determine therefore a positive reciprocal square matrix with n rows and n columns (where n is equal to the number of alternatives) called matrix of paired comparisons.

Below a detailed explanation of all steps of the procedure is proposed, taking as example the comparison of all five reporting systems with respect to sub-criterion 2.2.1 (number of indicators). From the paired comparisons, a 5 × 5 matrix is obtained (Table 7.11), where the values along the main diagonal correspond to 1, given that each alternative is equal to itself. Considering, for example, the comparison between C and E, it is possible to see that C is extremely preferred to E, since it has a lower number of indicators, and that the coefficient of dominance of E compared to C is none other than the reciprocal of that of C compared to E. The other comparisons can be interpreted in the same way.

Table 7.11 Matrix of paired comparisons—sub-criteria 2.2.1

At this point, the AHP model requires to:

• Standardise the matrix of paired comparisons, calculating the sum of each of the columns and dividing each element of the matrix by the value of the sum of the vector column to which that particular element belongs (Table 7.12);

  Table 7.12 Standardised matrix of paired comparison—sub-criteria 2.2.1

• Calculate the average of each row of the standardised matrix, obtaining in this way the vector of weights w (Table 7.13).

  Table 7.13 Vector of weights—sub-criteria 2.2.1

Before relying completely on the calculated weights, it is necessary to verify the consistency of the original matrix of paired comparison. AHP requires to:

• Multiply the matrix of the paired comparisons (not standardised) by the vector of weights w (Table 7.14);

  Table 7.14 Matrix of paired comparison multiplied by the vector of weights—sub-criteria 2.2.1

• Divide this vector by the vector of the weights w, to obtain the vector λ _max (Table 7.15);

  Table 7.15 Vector λ _max

• Calculate the average of the values that make up vector λ _max in order to obtain an average λ _max.

When the matrix is perfectly consistent, λ _max = n, i.e. to the number of alternatives that must be compared.

By calculating the average of the values that make up vector λ _max, we obtain a λ _max average equal to 5.27. If the matrix were perfectly consistent, λ _max = n = 5. In our case, this does not happen, but if we calculate the two indices specifically developed by Saaty, the Consistency Index (CI) and the Consistency Ratio (CR), we can consider it to be acceptable. Indeed if CR ≤ 0.10, the degree of inconsistency in the paired comparisons matrix is acceptable and it is therefore possible to assume that the priorities obtained are significant. If on the other hand CR > 0.10, it may be that serious inconsistencies exist in the paired comparisons, and as a consequence, the analysis may not bring relevant results.

In this case, with CR = 0.06, the matrix may be considered to be consistent and the weights calculated to be significant.²

$$ \mathrm{CI}=\left({\lambda}_{\max}\hbox{--} n\right)/\left( n-1\right)=0.07 $$

(7.3)

$$ \mathrm{CR}=\mathrm{CI}/1.12=0.06<0.10 $$

(7.4)