Skip to main content
SpringerLink
Log in

Identifying the implied: Findings from three differentiated replications on the use of security requirements templates

  • Published:
Empirical Software Engineering Aims and scope Submit manuscript

Abstract

Identifying security requirements early on can lay the foundation for secure software development. Security requirements are often implied by existing functional requirements but are mostly left unspecified. The Security Discoverer (SD) process automatically identifies security implications of individual requirements sentences and suggests applicable security requirements templates. The objective of this research is to support requirements analysts in identifying security requirements by automating the suggestion of security requirements templates that are implied by existing functional requirements. We conducted a controlled experiment in a graduate-level security class at North Carolina State University (NCSU) to evaluate the SD process in eliciting implied security requirements in 2014. We have subsequently conducted three differentiated replications to evaluate the generalizability and applicability of the initial findings. The replications were conducted across three countries at the University of Trento, NCSU, and the University of Costa Rica. We evaluated the responses of the 205 total participants in terms of quality, coverage, relevance and efficiency. We also develop shared insights regarding the impact of context factors such as time, motivation and support, on the study outcomes and provide lessons learned in conducting the replications. Treatment group, using the SD process, performed significantly better than the control group (at p-value <0.05) in terms of the coverage of the identified security requirements and efficiency of the requirements elicitation process in two of the three replications, supporting the findings of the original study. Participants in the treatment group identified 84 % more security requirements in the oracle as compared to the control group on average. Overall, 80 % of the 111 participants in the treatment group were favorable towards the use of templates in identifying security requirements. Our qualitative findings indicate that participants may be able to differentiate between relevant and extraneous templates suggestions and be more inclined to fill in the templates with additional support. Security requirements templates capture the security knowledge of multiple experts and can support the security requirements elicitation process when automatically suggested, making the implied security requirements more evident. However, individual participants may still miss out on identifying a number of security requirements due to empirical constraints as well as potential limitations on knowledge and security expertise.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Notes

  1. http://www.hl7.org/

  2. https://msdn.microsoft.com/en-us/magazine/cc163519.aspx

  3. http://go.ncsu.edu/secreqtemplatesstudy

  4. https://sites.google.com/a/ncsu.edu/csc515-software-security/

  5. http://www.isaca.org/cobit/pages/default.aspx

  6. http://csrc.nist.gov/publications/nistpubs/800-30-rev1/sp800_30_r1.pdf

  7. http://www.coso.org/ERM-IntegratedFramework.htm

  8. http://go.ncsu.edu/secreqtemplatesstudy

  9. http://agile.csc.ncsu.edu/iTrust/wiki/doku.php?id=requirements

  10. http://www.va.gov/vler/vlerdocs_userstories.asp

  11. http://www.cyclos.org/mobilebanking/

  12. https://readability-score.com/

  13. http://graphpad.com/quickcalcs/kappa1/?K=5

References

  • Alexander I (2003) Misuse Cases: Use Cases with Hostile Intent. IEEE Softw 20(1):58–66

    Article  Google Scholar 

  • Braz F, Fernandez EB, VanHilst M (2008) Eliciting security requirements through misuse activities. 4th International Conference on Trust, Privacy & Security in Digital Busines (TrustBus’08), Turin, Italy, September 1–5, 2008, pp 328–333

  • Carver J (2010) Towards reporting guidelines for experimental replications: a proposal. 1st International Workshop on Replication in Empirical Software Engineering Research (RESER) [Held during ICSE 2010], Cape Town, South Africa

  • Carver J, Jaccheri L, Morasca S (2010) A checklist for integrating student empirical studies with research and teaching goals. Empir Softw Eng 15:35–59

    Article  Google Scholar 

  • Carver J, Juristo N, Baldassarre M, Vegas S (2014) Replications of software engineering experiments. Empir Softw Eng 19(2):267–276

    Article  Google Scholar 

  • De Gramatica M, Labunets K, Massacci F, Paci F, Tedeschi A (2015) The role of catalogues of threats and security controls in security risk assessment: an empirical study with ATM professionals. 21st International Working Conference on Requirements Engineering: Foundation for Software Quality (REFSQ2015), Springer Verlag, pp 98–114

  • Fabian B, Gürses S, Heisel M, Santen T, Schmidt H (2010) A comparison of security requirements engineering methods. Requirements Engineering - Special Issue on RE’09: Security Requirements Engineering 15: 7–40

  • Firesmith DG (2004) Specifying Reusable Security Requirements. J Object Technol 3(1):15

    Article  Google Scholar 

  • Gray PH, Meister DB (2004) Knowledge sourcing effectiveness. Manag Sci 50(6):821–834

    Article  Google Scholar 

  • Haley CB, Laney R, Moffett JD, Nuseibeh B (2008) Security requirements engineering: a framework for representation and analysis. IEEE Trans Softw Eng 34(1):133–153

    Article  Google Scholar 

  • Ito Y, Washizaki H, Yoshizawa M, Fukazawa Y, Okubo T, Kaiya H, Hazeyama A, Yoshioka N, Fernandez E (2015) Systematic mapping of security patterns research. Plop 2015

  • Karpati P, Opdahl AL, Sindre G (2015) Investigating security threats in architectural context: experimental evaluations of misuse case maps. J Syst Softw 104:90–111. doi:10.1016/j.jss.2015.02.040. Elsevier Ltd

  • Kitchenham B, Charters S (2007) Guidelines for performing systematic literature reviews in software engineering. Technical Report EBSE-2007-01 School of Computer Science and Mathematics, Keele University

  • Lindsay RM, Ehrenberg ASC (1993) The design of replicated studies. Am Stat 47(3):217–228

    Google Scholar 

  • McCrum-Gardner E (2008) Which is the correct statistical test to use? Br J Oral Maxillofac Surg 46(1):38–41. doi:10.1016/j.bjoms.2007.09.002

    Article  Google Scholar 

  • McDermott J, Fox C (1999) Using abuse case models for security requirements analysis. In: Computer Security Applications Conference, pp 55–64

  • Mead NR, Houg ED, Stehney TR (2005) Security quality requirements engineering (SQUARE) methodology. Technical Report CMU/SEI-2005-TR-009 Software Engineering Institute, Carnegie Mellon University

  • Mellado D, Fernández-Medina E, Piattini M (2007) A common criteria based security requirements engineering process for the development of secure information systems. Comput Stand Interfaces 29(2):244–253

    Article  Google Scholar 

  • Mellado D, Blanco C, Sánchez LE, Fernández-Medina E (2010) A systematic review of security requirements engineering. Comput Stand Interfaces 32:153–165

    Article  Google Scholar 

  • Meneely A, Smith B, Williams L (2012) Appendix B: iTrust electronic health care system case study. Software and Systems Traceability, Springer Verlag, pp 425–438

  • Menzies T, Dekhtyar A, Distefano J, Greenwald J (2007) Problems with precision: a response to “Comments on ‘data mining static code attributes to learn defect predictors”. IEEE Trans Softw Eng 33(9):637–640

    Article  Google Scholar 

  • Riaz M, King J, Slankas J, Williams L (2014) Hidden in plain sight: automatically identifying security requirements from natural language artifacts. Requirements Engineering (RE 2014). Karlskrona, Sweden, pp 183–192

  • Riaz M, Slankas J, King J, Williams L (2014) Using templates to elicit implied security requirements from functional requirements − a controlled experiment. International Symposium on Empirical Software Engineering and Measurement (ESEM), Torino, Italy

  • Riaz M, Breaux T, Williams L (2015) How have we evaluated software pattern application? a systematic mapping study of research design practices. Inf Softw Technol 65:14–38

    Article  Google Scholar 

  • Riaz M, Elder S, Williams L (2016) Systematically developing prevention, detection, and response patterns for security requirements. 3rd International Workshop on Evolving Security and Privacy Requirements Engineering (ESPRE), Beijing, China

  • Schumacher M, Fernandez-Buglioni E, Hybertson D, Buschmann F, Sommerlad P (2006) Security Patterns: Integrating Security and Systems Engineering. John Wiley & Sons, Ltd., West Sussex

    Google Scholar 

  • Sindre G, Opdahl AL (2005) Eliciting security requirements with misuse cases. Requir Eng 10(1):34–44. doi:10.1007/s00766-004-0194-4

    Article  Google Scholar 

  • Suleiman H, Svetinovic D (2013) Evaluating the effectiveness of the security quality requirements engineering (SQUARE) method: a case study using smart grid advanced metering infrastructure. Requirements Engineering 18(3):251–279

  • Taubenberger S, Jürjens J, Yu Y, Nuseibeh B (2011) Problem analysis of it-security risk assessment methods – an experience report from the insurance and auditing domain. Future Challenges in Security and Privacy for Academia and Industry, pp 259–270

  • Taubenberger S, Jürjens J, Yu Y, Nuseibeh B (2013) Resolving vulnerability identification errors using security requirements on business process models. Inf Manag Comput Secur 21(3):202–223

    Article  Google Scholar 

  • Toval A, Nicolás J, Moros B, García F (2002) Requirements reuse for improving information systems security: a practitioner’s approach. Requir Eng 6(4):205–219

    Article  MATH  Google Scholar 

  • Viera AJ, Garrett JM (2005) Understanding interobserver agreement: the kappa statistic. Fam Med 37(5):360–363

    Google Scholar 

  • Walia GS, Carver JC (2009) A systematic literature review to identify and classify software requirement errors. Inf Softw Technol 51(7):1087–1109

    Article  Google Scholar 

  • Wen Y, Zhao H, Liu L (2011) Analysing security requirements patterns based on problems decomposition and composition. First International Workshop on Requirements Patterns (RePa), pp 11–20

  • Withall S (2007) Software requirement patterns. Microsoft Press

  • Wohlin C, Runeson P, Höst M, Ohlsson M, Regnell B, Wesslén A (2000) Planning. In: Basili VR (ed) Experimentation in software engineering: an introduction. Kluwer Academic Publishers, Norwell, MA

    Chapter  Google Scholar 

  • Yoshioka N, Washizaki H, Maruyama K (2008) A survey on security patterns. Progress in Informatics, Special Issue: The future of software engineering for security and privacy (5): 35–47

  • Yskout K, Scandariato R, Joosen W (2015) Do security patterns really help designers? Proc. of ICSE 2015. IEEE, pp 292–302

  • Zhang C, Budgen D (2012) What do we know about the effectiveness of software design patterns? IEEE Trans Softw Eng 38(5):1213–1231

    Article  Google Scholar 

Download references

Acknowledgments

This work is partially supported by NSA Science of Security lablet. Fabio Massacci is partially supported by the SESAR Joint Undertaking WP-E EMFASE Project. Christian Quesada-López and Marcelo Jenkins are supported by University of Costa Rica Project No. 834-B5-A18, and Ministry of Science, Technology and Telecommunications (MICITT). Special thanks to Patrick Francis and Patrick Morrison with their help in developing the study oracle. We are thankful to the Realsearch group for their collaboration and helpful comments.

Author information

Authors and Affiliations

  1. Department of Computer Science, North Carolina State University, 890 Oval Drive, Box 8206, Engineering Building II, Raleigh, NC, 27695-8206, USA

    Maria Riaz, Jason King, John Slankas & Laurie Williams

  2. University of Trento, Trento, Italy

    Fabio Massacci

  3. University of Costa Rica, San José, Costa Rica

    Christian Quesada-López & Marcelo Jenkins

Authors
  1. Maria Riaz
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jason King
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. John Slankas
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Laurie Williams
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Fabio Massacci
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Christian Quesada-López
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Marcelo Jenkins
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Maria Riaz.

Additional information

Communicated by: Andreas Zeller

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Riaz, M., King, J., Slankas, J. et al. Identifying the implied: Findings from three differentiated replications on the use of security requirements templates. Empir Software Eng 22, 2127–2178 (2017). https://doi.org/10.1007/s10664-016-9481-1

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10664-016-9481-1

Keywords

Search

Navigation