To validate the SRPs expressed in RAST and their application process contribution we have conducted an observatory study. Its objectives were (i) to understand correctness and (ii) usability the SRP application, (ii) to compare understandability of the SRP application of the participants with the ISSRM background against participants without ISSRM background.
5.1 Observatory Study Design
Participants. We have invited six individuals with the software engineering background. Three participants (i.e., group A) had the IS-security background as they were working in the field of enterprise security and had prior knowledge of the ISSRM concepts. Other three participants (i.e., group B) had no information systems security background, but nevertheless they were practitioners working in the software engineering companies.
Design. Firstly, the participants were given the introductory lecture. Secondly, they participants were given a Secure Tropos model and were asked to derive the security requirements using the SRPs. Finally, the participants filled the questionnaire on the usability of the SRPs and their application process. Each participant took approximately 3 h to complete the process.
Treatment. The lecture included an introduction of the ISSRM domain model and security risk management process (see Sect. 2.1), RAST (see Sect. 2.2), SRPs (see Sect. 3) and their application process (see Sect. 4). The lecture was concluded with an SRP application demo.
SRP application task focussed on the pattern application process (see Sect. 4). Participants were given a model described in Sect. 3 and were requested to identify SRP occurrences as well as derive security requirements. None of the participants applied all the SRPs (see Table 1), since we limited their participation to three hours.
Questionnaire. Once the SRP application was completed, participants filled the questionnaire on the usability of the RAST process, SRPs and SRP application process. Specifically in included questions on easiness, satisfaction, and understandability of used artefacts.
5.2 Threats to Validity
The following threats to validity should be taken into account:
The number of participants is rather small (only six participants) thus the sample may not be accurate. The results might differ if we were able to attract more participants. However they all were practitioners working in the field of software security engineering.
Given treatment could influence the received results. When applying SRPs participants had some questions, and we provided then with the answers. However otherwise they would have difficulty to complete the given task.
Each participant conducted his task individually. If participants had opportunity to discuss and to learn from each the result potentially would be different.
Each participant applied different SRPs. Ideally all of the participants would have to complete all SRPs for better result. However, we were limited by the time constraints (three hours per participant).
Participants had a varying level of prior knowledge of ISSRM domain and security risk management. Having participants with the same knowledge could potentially deliver more reliable result. We tried to mitigate this threat by provided introductory lecture.
The majority of the participants implemented the models using an online drawing tool. Implementing the models by hand or other modelling tool method could impact the modelling outcome.
The participants were not told that they were expected to perform in a certain way or that a specific result was expected from them. Stating expectations upfront would impact the overall performance of the participants. The performance could be enhanced in case the participant would want to perform according the expectations. Or the participant could suffer from a type of performance anxiety and his result would be negatively affected.
Participant had prior acquaintance with the conductor of the observatory study and the first author of this paper. If no prior acquaintance would occur participants could not ask the same questions or perform in the same manner they would perform to another individual. However this acquaintance was the way to involve participants in the evaluation.
Some SRPs were easier identified in the model comparatively to other SRPs. If all the SRPs would be identical in terms of identification ease, the result could be different. Making an SRP easier or harder to identify results in the pattern application process becoming automatically easier of harder to perform.
All the above threats had a certain effect to the overall results. We assume that in case of a more extensive study with a greater number of participants and different design, different outcomes might be received.
5.3 Observatory Study Results
Correctness of the Participant Models. Correctness of each SRP application is defined through the number of errors identified in the resulting model (i.e., the lower number indicates better model correctness). Errors are divided to two categories phrasing and modelling errors. Phrasing errors describe any error in regards to the phrasing of any of the constructs (e.g. labels of goals, plans, and etc.). Modelling errors describe errors performed in the modelling of each concept. Modelling errors include using wrong constructs when linking assets, risk components, security countermeasures and similar. Additionally, modelling mistakes also include incorrect colouring of the constructs (as the colour here brings the semantic difference between security risk concepts). Both types of errors are discovered by comparing the participants’ models with the models prepared by the first author of this paper.
Table 1 presents the result of the model correctness. It was observed that Participant 2 has made the least amount of errors compared to the other participants. It is also important is to point out that the majority of errors done by Participant 1 were rather minor in comparison to phrasing errors of the other participants. In general the majority of the errors are phrasing errors. This could be explained by the fact that the modelling language does not provide explicit guidelines on how to name the constructs during modelling.
Understandability. As mentioned in the design description, the participants were divided to two groups – group A and group B – based on their previous experience with the security engineering. The results show that participants of group A were able to apply and comprehend the used patterns as well as the pattern application process. Participant of group B were able moderately to apply and to comprehend SRPs. Group A correctly executed all the pattern application steps. Nonetheless mistakes were made in phrasing and resource decomposition (as discussed above). But they performed all the tasks in a rather reasonable time and were confident in their results.
Group B completed the SRP application process with a moderate correctness. Similar to group A, group B also made mistakes in phrasing and modelling. Furthermore, noticeable difference in the results was the level of confidence in the results of the application process. Participants of group B were notably less confident than the participants of group A in their results. As conclusion we observed that the information systems security experience had some impact and helped better contributed to the understandability of the SRPs application.
Discussion and Concluding Remarks. We draw out the concluding remarks based on our observations and the participant responses marked in the questionnaire. All participants completed the application of at least one pattern. Mistakes were observed in the phrasing and modelling of various assets of the models. In comparison less mistakes were made in modelling rather than phrasing. All the participants understood the proposed SRP representations. The pattern application process was according to the majority of the participants moderately easy to be applied. RAST affects the overall process in a moderate level.
The fact that both groups were able to complete the tasks assigned, demonstrated that the process is useable as a starting point to derive security requirements in a goal-oriented environment. The easiest part in the application process according to the majority of the participants was the pattern identification and asset alignment. The hardest step to be applied by the majority of the participants was security requirement introduction and extracted model re-integration.
Having background knowledge in IS security affects the process during the first applications and speeds up moderately the security requirement derivation process. Prior knowledge of an agent-oriented language in combination ISSRM affects rather positively the outcome of modelling. Participant that had no prior knowledge were less confident about their results. The following lessons are learnt:
Application of the SRPs helps to construct rather correct security models and derive appropriate security requirements. The major modelling mistakes are made due to the lack of guidance from the modelling language application.
The SRP application guidelines are rather understandable and moderately usable by their users. However, a priori experience in security engineering helps to see the method purpose. Potentially some security engineering training could help to improve method application.