Practical challenges of requirements prioritization based on risk estimation

Abstract

Requirements prioritization and risk estimation are known to be difficult. However, so far, risk-based requirements prioritization has not been investigated empirically and quantitatively. In two quantitative experiments, we explored practical challenges and needs of risk estimations in general and of our method MOQARE specifically. In the first experiment, ten students made individual estimations. In the second one, twenty-four students estimated risks in seven moderated groups. The students prioritized the same requirements with different methods (risk estimation and ranking). During the first experiment, we identified factors which influence the quality of the prioritization. In the second experiment, the results of the risk estimation could be improved by discussing risk estimations in a group of experts, gathering risk statistics, and defining requirements, risks and prioritization criteria more tangibly. This first quantitative study on risk-based requirements prioritization helps to understand the practical challenges of this task and thus can serve as a basis for further research on this topic.

Annex A: Data and Data Analysis

This annex for Experiment 1 and 2 describes the quantitative analysis of the variables defined in Section 3. For each variable, the results from both experiments and all methods are presented together. Their interpretation, especially how we believe that these variables have been influenced by the influencing factors, is discussed in Section 6.

1.1 A.1 Time Consumption

Method 1 demands to determine only two values (group and priority) for each of the countermeasures. Method 2 demands the estimation of two probabilities and two damages per countermeasure.

In Experiment 1, the time needed for Method 2 (risk estimation) is significantly higher than in Method 1 (ranking). The average time needed for Q1 was 6.6 min for those 7 participants who noted it. Q2 took an average of 31.8 min. In Experiment 2, the time need averaged over those groups who performed this method first, was 17.5 min for Method 1 and 37.3 min for Method 2. (We count only these groups, because of the learning effect observed.)

From these numbers, we calculated the time need per countermeasure (Table 5) and also the time need per estimation (Table 6.), as the number of estimations per countermeasure in Method 2 depends on the number of misuse cases.

1.2 A.2 Priorities

The resulting priorities of the countermeasures varied widely among the participants and groups in both experiments, for both methods, as can be seen from Tables 15, 16, 17, 18. This means that they differ greatly about the importance of the countermeasures. The same countermeasure could have the highest priority (1) for one participant/ group and the lowest for another. This was the case even in Method 1, where the results were transparent to and manipulable by the estimators, while in Method 2 the lacking transparency and indirect manipulability of the priorities could possibly lead to results which are unexpected by the estimators.

Table 15 Priorities resulting from Experiment 1 with Method 1 (“1” standing for the most important one): The “1” is row “R1” and column “1” means that according to the results of participant 1, Countermeasure R1 is the most important one

Table 16 Priorities resulting from experiment 1 with method 2 (“1” indicating the most important one)

Table 17 Priorities resulting from experiment 2 with method 1 (“1” indicating the most important one)

Table 18 Priorities resulting from experiment 2 with method 2 (“1” indicating the most important one)

In Method 1, the averages of the priorities lie between 3.0 and 6.4 for the individual countermeasures. In Method 2, the averages are between 3.1 and 7.2. In Experiment 2, all groups agreed that R3 is one of the least important: In Method 1, all seven groups gave R3 the lowest priority 7, while in Method 2, this was the case for five groups, once it received priority 6 and once 5. The priority averages in Method 1 (not counting R3), vary from 2.57 to 5.00, and in Method 2 from 2.29 to 4.21.

We are sure that these wide ranges are not caused by the misunderstanding whether “1” stands for the highest priority or the lowest. In Method 1, the priority 1 countermeasure for all participants was found in the “high benefit” group. In Method 2, the priorities were determined by us, based on the calculated benefits.

1.3 A.a Standard Deviation of Priorities

The standard deviations s found among the ten estimations of the nine priorities in Experiment 1 are shown in Table 7 and also the seven estimations found among the seven priorities in Experiment 2 in Table 8. The differences of the standard deviations between the methods in the same experiment are very low and statistically not significant. The standard deviations in Experiment 2 are lower than in Experiment 1 because fewer countermeasures were prioritized, but also when divided by the average priority (which is (n + 1)/2), in Experiment 2 the standard deviation is lower (see Table 9).

1.4 A.b Ease of Use

The ease of use of each method as rated by the participants is shown in Table 10. We attribute points to the answers: Very easy =2 points, Easy =1, Undecided =0, Difficult =−1, Very difficult =−2.

1.5 A.c Participants Expect Their Estimations to Be Realistic

In Experiment 1, immediately after the estimations, but before they knew the resulting priorities, we asked the participants whether they expected to have made reasonable, realistic and useful estimations (question Q4b, see Table 11). In Experiment 2, they were asked whether they believe that their results were realistic. At this point of time, they knew the priorities. Points were attributed to the answers: very = 2 points, rather =1, undecided = 0, rather not =−1, not at all = −2 points.

It is interesting to note that in Experiment 2, the average points for the probability and damage estimations were 0.13, i.e. lower than for the priorities (but not statistically significantly due to high variations). This means that some participants trusted the method to deliver priorities which are more realistic than the risk estimations on which they are based.

1.6 A.d Accuracy of the Results

Accuracy here means that after the experiment the participants considered the countermeasure priorities which resulted from their risk estimations to be plausible and reflecting their views. In Experiment 1, this question was asked in the post-test session 1 week after the experiment, in Experiment 2, the question was asked during the experiment session directly after the estimations.

In Experiment 1, nine participants answered this question (free-text answers). Four were in favour of Method 1, two wrote, that their first impression was that Method 1 delivered the most plausible results, because they corresponded to their intuitive priorities, but as Method 2 was a systematic method, probably this method should provide the best results. Another participant wrote that all methods delivered plausible as well as less plausible results for the different countermeasures. The last participant wrote that they were all plausible: Method 1 reflected his own perception, neglecting risk and cost. Method 2 was plausible as well, taking into account risk and cost.

In Experiment 2, Method 1 got an average of 1.29 points as a result to this question; Method 2 got 0.13 points on a scale between −2 and +2 (very well = 2 points, rather yes = 1, undecided = 0, rather not =−1, very badly = −2 points). This difference is statistically significant.

1.7 A.e Frequency of Naming a Misuse Case or Countermeasure as Especially Uncertain

The participants were asked to name the countermeasures or misuse cases for which they considered their estimated values especially uncertain. We counted how often a certain countermeasure (in Method 1) or a certain misuse case (in Method 2) was named (Variable e). To compare the methods and countermeasures, we calculated the average frequency with which a countermeasure or misuse case was named here, averaged over all countermeasures/ misuse cases. We did so, because the number of countermeasures and misuse cases was not the same in the two experiments and the methods applied. The results are summarized in Table 12.

In Experiment 1, regarding Method 1, R8 was never named and R3 only once. R5 was mentioned twice, R1, R2, R4 and R9 three times, R7 four times and R6 five times by the ten participants. Concerning Method 2, eight out of ten participants named specific misuse cases, while six additionally or instead said that they were uncertain about practically all of them. If we take the latter group literally, each misuse case was named with the average frequency 0.7. If we do not count the participants stating they were uncertain concerning all misuse cases, the numbers are about 0.2 for the probability estimations as well as for the damage estimations.

In Experiment 2, regarding Method 1, R1 was named seven times (by the 18 participants who answered to this question), R6 and R7 four times, R2, R4 and R5 three times and R3 only once. Except for R1, which seems to be especially difficult to judge, and R3, which caused almost no irritation, most countermeasures seem to be equally difficult. Few correlations are seen among the answers of members of the same group. Only once, all three group members agreed that R1 was difficult, and three times two of three or four group members agreed about the same countermeasure.

In Experiment 2, concerning the probability estimation in Method 2, Misuse Cases 1–3 were named four times (by the 18 participants who answered this question), Misuse Case 4 only once, Misuse Case 5 nine times and Misuse Case 6 eight times. Only once, all three group members agreed about the same misuse case: for Misuse Case 5 they all found probability estimation difficult. Five times, two group members agreed about the same misuse case.

In Experiment 2, concerning the damage estimation in Method 2, the misuse cases were named with the following frequencies (by the 18 participants who answered this question): Misuse Case 6 nine times, 1 and 4 seven times each, 3 five times, 5 four times, and 2 only once. Only once all three group members agreed concerning 3. Seven times, two group members considered the same misuse case’s damage estimation to be difficult. This means an average frequency for a misuse case of 0.3. These numbers are approximately the same as for the probability estimation.

The differences observed between methods (during the same experiment) and during different experiments for the same method are statistically significant.

1.8 A.3 Influence of Statistics

In Experiment 1, we tested the influence of statistics provided to the estimators. In Q5d, 8 out of 9 participants now clearly attributed different probabilities (reference risk) to the two security misuse cases, usually much higher ones, see Table 13 and Table 14. One participant wrote that he estimated the same probabilities as before (0.5% and 0.1%), but we doubt whether they were really derived from the statistics, as they differ too much from the estimations of the other participants.

When questioned whether the statistics facilitated the probability estimation, we received free-text answers. Some of them were: “They definitely were helpful. One feels more certain, thanks to this information.” Others were more sceptical: “I would say that the statistics have strongly influenced my estimations. However, I wonder how similar the systems of these companies are to the reference system in the case study. Only when this is known, one can say whether the high estimated value is justified. I think that I still do not have enough information to deliver a good estimation.” All together, 4 participants out of 9 wrote that the statistics were helpful, while one wrote they did not influence the estimation and four wrote they influenced the estimations, but they still were sceptical whether the estimated values were exact.

We expected that when providing the participants with several statistics, the standard deviation (relative to the average) of their estimations to decrease (Variable a). As one can see in Table 13 and Table 14, the estimated probabilities still differed among the participants. This was to be expected because four statistics were given, which did not apply to exactly the same environment as the case study. Therefore, interpretations and adaptations were necessary. The coefficient of variation became less than half by using the statistics.