Abstract
The quality of requirements is typically considered as an important factor for the quality of the end product. For traditional up-front requirements specifications, a number of standards have been defined on what constitutes good quality : Requirements should be complete, unambiguous, specific, time-bounded, consistent, etc. For agile requirements specifications, no new standards have been defined yet, and it is not clear yet whether traditional quality criteria still apply. To investigate what quality criteria for assessing the correctness of written agile requirements exist, we have conducted a systematic literature review. The review resulted in a list of 16 selected papers on this topic. These selected papers describe 28 different quality criteria for agile requirements specifications. We categorize and analyze these criteria and compare them with those from traditional requirements engineering. We discuss findings from the 16 papers in the form of recommendations for practitioners on quality assessment of agile requirements. At the same time, we indicate the open points in the form of a research agenda for researchers working on this topic .
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
References
Auger, C. P. (1994) Information sources in grey literature. Bowker-Saur.
Beck, K. (2002). Test driven development: By example. Boston, MA: Addison-Wesley Longman Publishing Co. Inc.
Beck, K., Beedle, M., van Bennekum, A., Cockburn, A., Cunningham, W., Fowler, M. et al. (2001). Manifesto for agile software development. http://agilemanifesto.org. Last visited April 16, 2015.
Belsis, P., Koutoumanos, A., & Sgouropoulou, C. (2014). PBURC: A patterns-based, unsupervised requirements clustering framework for distributed agile software development. Requirements Engineering, 19(2), 213–225.
Bhasin, S. (2012). Quality assurance in agile: A study towards achieving excellence. In AGILE India (AGILE INDIA) (pp 64–67).
Bjarnason, E., Unterkalmsteiner, M., Engstrm, E., & Borg, M. (2015). An industrial case study on test cases as requirements. In C. Lassenius, T. Dingsyr, & M. Paasivaara (Eds.) Agile Processes, in Software Engineering, and Extreme Programming, Lecture Notes in Business Information Processing (Vol. 212, pp. 27–39) Springer International Publishing. doi:10.1007/978-3-319-18612-2_3.
Brereton, P., Kitchenham, B. A., Budgen, D., Turner, M., & Khalil, M. (2007). Lessons from applying the systematic literature review process within the software engineering domain. Journal of Systems and Software, 80(4), 571–583.
Cornelissen, B., Zaidman, A., van Deursen, A., Moonen, L., & Koschke, R. (2009). A systematic survey of program comprehension through dynamic analysis. IEEE Transactions on Software Engineering, 35(5), 684–702.
Davis, A., & Hickey, A. (2009). A quantitative assessment of requirements engineering publications 1963–2008. In M. Glinz & P. Heymans (Eds.), Requirements engineering:Foundation for software quality (Vol. 5512, pp. 175–189). Lecture Notes in Computer Science, Berlin Heidelberg: Springer.
Davis, A., Overmyer, S., Jordan, K., Caruso, J., Dandashi, F., Dinh, A. et al. (1993). Identifying and measuring quality in a software requirements specification. In Int’l Software Metrics Symposium (pp. 141–152). Baltimore (MD).
De Lucia, A., & Qusef, A. (2010). Requirements engineering in agile software development. Journal of Emerging Technologies in Web Intelligence, 2(3), 309–326.
Desharnais, J. M., Kocaturk, B., & Abran, A. (2011). Using the cosmic method to evaluate the quality of the documentation of agile user stories. In Joint Conference of the 21st Int’l Workshop on Software Measurement and 6th Int’l Conference on Software Process and Product Measurement (IWSM-MENSURA), Nara, Japan (pp. 269–272).
Dingsøyr, T., Dyba, T., & Abrahamsson, P. (2008). A preliminary roadmap for empirical research on agile software development. In AGILE Conference, Toronto (ON) (pp. 83–94).
Dumas-Monette, JF., & Trudel, S. (2014). Requirements engineering quality revealed through functional size measurement: An empirical study in an agile context. In Joint Conference of the International Workshop on Software Measurement and the International Conference on Software Process and Product Measurement (IWSM-MENSURA), Rotterdam, The Netherlands (pp. 222–232).
Duncan, R. (2001). The quality of requirements in extreme programming. CrossTalk, June 19, pp. 22–31.
Dybå, T., & Dingsøyr, T. (2008). Empirical studies of agile software development: A systematic review. Information and Software Technology, 50(9–10), 833–859.
Eberlein, A., & Leite, J. (2002). Agile requirements definition: A view from requirements engineering. In Proceedings of the International Workshop on Time-Constrained Requirements Engineering (TCRE02), Essen, Germany (pp. 4–8).
Ernst, N. A., & Murphy, G. (2012). Case studies in just-in-time requirements analysis. Int’l Workshop on Empirical Requirements Engineering (pp. 25–32). Chicago (IL): IEEE.
Farid, W., & Mitropoulos, F. (2013). Norplan: Non-functional requirements planning for agile processes. In Proceedings of IEEE Southeastcon, Jacksonville, FL (pp. 1–8).
Firesmith, D. (2004). Generating complete, unambiguous, and verifiable requirements from stories, scenarios, and use cases. Journal of Object Technology, 3(10), 27–40.
Gottesdiener, E., & Gorman, M. (2010). Slicing requirements for agile success. Better Software, 2010(04), 16–21.
Grau, R., Lauenroth, K., Bereza, B., van Veenendaal, E., & van der Zee, S. (2014). Requirements engineering and agile development - collaborative, just enough, just in time, sustainable. IREB.
Heck, P., & Zaidman, A. (2014). A quality framework for agile requirements: A practitioner’s perspective. Technical Report TUD-SERG-2014-006, Software Engineering Research Group, Delft University of Technology.
Heck, P., & Zaidman, A. (2016). A framework for quality assessment of just-in-time requirements: The case of open source feature requests. Requirements Engineering (pp. 1–21).
Huo, M., Verner, J., Zhu, L., & Babar, M. (2004). Software quality and agile methods. In Int’l Computer Software and Applications Conference (Vol.1, pp. 520–525). doi:10.1109/CMPSAC.2004.1342889.
IEEE. (1998). IEEE recommended practice for software requirements specifications. IEEE Std 830-1998.
Inayat, I., Salim, S. S., Marczak, S., Daneva, M., & Shamshirband, S. (2014). A systematic literature review on agile requirements engineering practices and challenges. Computers in Human Behavior, 51, 915–929.
Kabbedijk, J., Bezemer, C. P., Jansen, S., & Zaidman, A. (2015). Defining multi-tenancy: A systematic mapping study on the academic and the industrial perspective. Journal of Systems and Software, 100, 139–148.
Kassab, M. (2014). An empirical study on the requirements engineering practices for agile software development. In 40th EUROMICRO Conference on Software Engineering and Advanced Applications (SEAA), Verona, Italy (pp. 254–261).
Kitchenham, B., & Charters, S. (2007). Guidelines for performing systematic literature reviews in software engineering. Techical Report, EBSE-2007-01.
Kitchenham, B. A. (2004). Procedures for performing systematic reviews. Technical Report.
Lee, C., Guadagno, L., & Jia, X. (2003). An agile approach to capturing requirements and traceability. In Proceedings of the 2nd International Workshop on Traceability in Emerging Forms of Software Engineering, Montreal, Canada (pp. 17–23).
Leffingwell, D. (2011). Agile software requirements: Lean requirements practices for teams, programs, and the enterprise (1st ed.). Boston: Addison-Wesley Professional.
Liskin, O., Pham, R., Kiesling, S., & Schneider, K. (2014a). Why we need a granularity concept for user stories. In G. Cantone & M. Marchesi (Eds.) Agile Processes in Software Engineering and Extreme Programming, Lecture Notes in Business Information Processing (Vol. 179, pp. 110–125). Berlin: Springer International Publishing.
Liskin, O., Schneider, K., Fagerholm, F., & Münch, J. (2014b). Understanding the role of requirements artifacts in kanban. In Proceedings of the 7th International Workshop on Cooperative and Human Aspects of Software Engineering, Hyderabad, India, CHASE 2014 (pp. 56–63).
Melnik, G., Maurer, F., & Chiasson, M. (2006). Executable acceptance tests for communicating business requirements: Customer perspective. In Agile Conference, 2006 (pp. 12–46). doi:10.1109/AGILE.2006.26.
North, D. (2006). Behavior modification. Better Software, 2006, 4–12.
Paetsch, F., Eberlein, A., & Maurer, F. (2003). Requirements engineering and agile software development. In Int’l Workshop on Enabling Technologies: Infrastructure for Collaborative Enterprises, Linz, Austria (pp. 308–308).
Patel, C., & Ramachandran, M. (2009). Story card maturity model (smm): A process improvement framework for agile requirements engineering practices. JSW, 4(5), 422–435.
Power, K. (2014). Definition of ready: An experience report from teams at cisco. In G. Cantone, M. Marchesi (Eds.) Agile processes in software engineering and extreme programming. Lecture Notes in Business Information Processing (Vol. 179, pp. 312–319). Berlin: Springer International Publishing.
Ramesh, B., Cao, L., & Baskerville, R. (2010). Agile requirements engineering practices and challenges: An empirical study. Information Systems Journal, 20(5), 449–480.
Runeson, P., Host, M., Rainer, A., & Regnell, B. (2012). Case study research in software engineering: Guidelines and examples. New York: Wiley.
Sfetsos, P., & Stamelos, I. (2010). Empirical studies on quality in agile practices: A systematic literature review. In Seventh International Conference on the Quality of Information and Communications Technology (QUATIC), Porto, Portugal (pp. 44–53).
Sillitti, A., & Succi, G. (2005). Engineering and managing software requirements, Chapter 14. In A. Aurum & C. Wohlin (Eds.), Requirements engineering for agile methods (pp. 309–326). Berlin: Springer.
Srinivasan, J., & Lundqvist, K. (2009). Using agile methods in software product development: A case study. In Sixth International Conference on Information Technology: New Generations. ITNG ’09, Las Vegas, NV (pp. 1415–1420).
Staples, M., & Niazi, M. (2007). Experiences using systematic review guidelines. Journal of Systems and Software, 80(9), 1425–1437.
Wake, B. (2003). INVEST in good stories, and SMART tasks. http://xp123.com/articles/invest-in-good-stories-and-smart-tasks/. Accessed December, 2015.
Wieringa, R., Maiden, N., Mead, N., & Rolland, C. (2005). Requirements engineering paper classification and evaluation criteria: A proposal and a discussion. Requirements Engineering, 11(1), 102–107.
Wohlin, C. (2014). Guidelines for snowballing in systematic literature studies and a replication in software engineering. In Proceedings of the International Conference on Evaluation and Assessment in Software Engineering (EASE), London, UK (pp. 38:1–38:10).
Author information
Authors and Affiliations
Corresponding author
Appendix: Details of selected papers on quality criteria for agile requirements
Appendix: Details of selected papers on quality criteria for agile requirements
This appendix contains detailed information on each of the 16 selected papers. We repeat author, title and venue and include number of pages. We provide a short summary of the paper based on the paper abstract. We briefly explain the relevance of the paper for quality criteria for agile requirements (in italic) and we sum up each of the quality criteria mentioned in the paper. For an overview of all quality criteria from all papers, see Fig. 2.
[ID] Reference | Title and Venue | #pp. | Summary and Relation to Quality Criteria |
|---|---|---|---|
PBURC: A Patterns-Based, Unsupervised Requirements Clustering Framework for Distributed Agile Software Development | 13 | This paper presents a patterns-based, unsupervised requirements clustering framework, which makes use of machine-learning methods for requirements validation, being able to overcome data inconsistencies and determine appropriate requirements clusters for the definition of software development sprints. | |
(Requirements Engineering Journal) | [B14] is of low relevance for quality criteria as it focuses on a clustering algorithm for requirements. | ||
Quality criteria: priority. | |||
The Quality of Requirements in Extreme Programming | 4 | This paper describes and evaluates the quality of requirements generated by using XP and discusses how the XP process can assist or hinder proper requirements engineering. | |
(Journal of Defense Software Engineering) | [D01] compares eXtreme Programming to a list of quality criteria for traditional requirements from Davis et al. (1993). | ||
Quality criteria: priority, product version, verifiable, concise, unambiguous. | |||
Using the COSMIC Method to Evaluate the Quality of the Documentation of Agile User Stories (IWSM-MENSURA) | 4 | In the research reported here, the COSMIC method is used to analyze and report on the quality of the documentation of user stories. The functional size of evolving requirements can be measured with COSMIC measurement method. To support this measurement activity, the quality of the documentation is important to be interpreted correctly. | |
See [D14]. | |||
Requirements Engineering Quality Revealed through Functional Size Measurement: An Empirical Study in an Agile Context (IWSM-MENSURA) | 11 | This paper reports preliminary results related to a software development organization. The functional size of the software was measured and compared with development and measurement effort, taking into account the quality rating of requirements. The results led to recommendations for the organization and recommendations for planning any software measurement project. | |
Both [D11] and [D14] focus on size measurement of agile requirements (using COSMIC). They claim that quality of those requirements is a necessary condition for accurate estimation. Quality in this sense mainly means completeness: descriptions of functional processes and data modeling artifacts are needed to base the COSMIC size measurements on. [D14] contains a case study that also highlights other quality issues related to size measurement: user story names should at least contain the name of the functional process it maps to in order to be correctly measured; it is expected that each user story be mapped to only one functional process; consistent terminology should be used for data groups/objects. | |||
Quality criteria: functional processes, data modeling artifacts, functionality in story name, small/simple, consistent/defined terminology. | |||
Requirements Engineering in Agile Software Development (Journal of Emerging Technologies in Web Intelligence) | 9 | This paper discusses problems concerned with requirements engineering in agile software development processes and suggests some improvements to solve some challenges caused by large projects. The paper also discusses the requirements traceability problem in agile software development. | |
[DQ10] is a general overview paper on requirements engineering for agile methods. The focus is not specifically on quality, but [DQ10] mentions some criteria: non-functional requirements should not be forgotten, requirements should be in the language of the customer, requirements should not be too complex, requirements should be forward traceable (i.e., to source code and tests). | |||
Quality criteria: non-functional requirements, small/simple, customer language, forward traceable. | |||
Generating Complete, Unambiguous, and Verifiable Requirements from Stories, Scenarios, and Use Cases | 13 | This paper shows how to transform incomplete and vague stories, scenarios, and use cases into a proper set of textual requirements. | |
(Journal of Object Technology) | [F04] promotes a template for writing textual requirements: ID, trigger, precondition, action, postcondition. | ||
Quality criteria: unique ID, template, unambiguous. | |||
NORPLAN: Non-functional Requirements Planning for Agile Processes (IEEE Southeastcon) | 8 | This research proposes project management and requirements quality metrics. NORPLAN proposes two prioritization schemes, Riskiest-Requirements-First and Riskiest-Requirements-Last, for planning release and sprint cycles using a risk-driven approach. The approach is validated through visual simulation and a case study. | |
[FM13] focuses on non-functional requirements planning for agile processes. It contains a table with agile requirements quality metrics. However, these metrics are mainly process-related, except for ambiguity. | |||
Quality criteria: non-functional requirements, unambiguous. | |||
Slicing Requirements for Agile Success | 8 | This paper presents different options for slicing user stories. | |
(Better Software Magazine) | [GG10] does not specifically focus on quality, but presents a way of slicing (splitting) user stories. By explaining the ways of slicing, they also mention six important elements of a user story: user roles, actions, data objects, business rules, interfaces, quality attributes (i.e., non-functional requirements). | ||
Quality criteria: six elements of a user story. | |||
A Quality Framework for Agile Requirements: A Practitioner’s Perspective (TU Delft Computer Science Report) | 11 | This paper presents a quality framework for informal verification of agile requirements and instantiates it for feature requests in open source projects. The framework was derived based on an existing framework for traditional requirements specifications, literature about agile and open source requirements and the authors’ experience with agile and open source requirements. | |
[HZ14] presents a framework for agile requirements quality criteria, which is instantiated for feature requests in open source projects and user stories. It presents a large number of quality criteria. The use of a template is indirectly also promoted because they claim that a tool should be used to store the requirements. This tool, e.g., an issue tracker, would have predefined fields for each requirement, thereby functioning like a template. | |||
Quality criteria: priority, unique ID, product version, acceptance criteria, rationale, template, only relevant comments, no contradiction/conflict, INVEST, consistent/defined terminology, correct language, forward traceable, no duplicates, navigable and explained links. | |||
An Agile Approach to Capturing Requirements and Traceability | 7 | This paper presents a tool-based approach that provides for the implicit recording and management of relationships between conversations about requirements, specifications, and design decisions. | |
(Workshop on Traceability in Emerging Forms of Software Engineering) | [L03] is of low relevance for our research question as it focuses on a tool for traceability. However, it mentions forward traceability (e.g., to tests and source code) as an important quality aspects of agile requirements. | ||
Quality criteria: forward traceable. | |||
Why We Need a Granularity Concept for User Stories (XP) | 16 | This paper investigates Expected Implementation Duration of a user story as a characteristic of granularity by conducting a study with software engineering practitioners. Many developers have experienced certain problems to occur more often with coarse user stories. The findings emphasize the importance to reflect on granularity when working with user stories. | |
[L14a] and [L14b] both focus on the usefulness of Expected Implementation Duration (EID) as a quality criterion for user stories. They observe that user stories should be kept small in terms of this EID. | |||
Quality criteria: small/simple. | |||
Understanding the Role of Requirements Artifacts in Kanban (Workshop on Cooperative and Human Aspects of Software Engineering) | 8 | This paper explores how to utilize requirements artifacts effectively, what their benefits and challenges are, and how their scope granularity affects their utility. It studies a software project carried out in the University of Helsinki. The requirements artifacts were investigated and then developers and the customer were interviewed about their experiences. | |
See [L14a]. | |||
Definition of Ready: An Experience Report from Teams at Cisco (XP) | 8 | This paper presents an example of definition of ready used by agile teams in Cisco. These teams have developed three levels of ready that apply for user stories, sprints and releases. The paper describes how definition of ready provides a focus for backlog grooming, and some consequences of not meeting definition of ready. The paper finishes with perspectives from different roles in the organization. | |
[P14] gives an example of the definition of ready (when are user stories ready to start implementing them?) for Cisco teams. This list of criteria mainly contains process-related items, but also some product-related: non-functional requirements should be identified, acceptance criteria should be identified, dependencies should be identified, size should have been estimated, team knows what it will mean to demo the user story (i.e., verifiable). | |||
Quality criteria: non-functional requirements, acceptance criteria, dependencies, size, verifiable. | |||
Story Card Maturity Model (SMM): A Process Improvement Framework for Agile Requirements Engineering Practices (Journal of Software) | 14 | This paper describes an ongoing process to define a suitable process improvement model for story cards based requirement engineering at agile software development environments: the SMM (based on CMM). It also presents how the identified areas of improvement from assessment can be mapped with best knowledge based story cards practices for agile software development environments. | |
[PR09] focuses on story cards. The appendix contains a list of guidelines for story cards, unfortunately without a detailed description of each guideline. The guidelines are placed in a maturity framework. | |||
Quality criteria: priority, unique ID, non-functional requirements, acceptance criteria, story card structure, no contradiction/conflict, concise, INVEST, consistent/defined terminology, unambiguous. | |||
Using Agile Methods in Software Product Development: A Case Study (ITNG) | 6 | This paper presents an in-depth case study of agile methods adoption in a software product development firm. Using a mix of interviews, observation and archival data, the evolution of agile adoption within the firm is reconstructed. | |
[SL09] is a short case study of the use of agile software development in a small company. They do not specifically mention requirements quality criteria, but they observe some problems that occurred because of the ambiguity of the requirements. | |||
Quality criteria: unambiguous. | |||
Requirements Engineering for Agile Methods (Engineering and Managing Software Requirements) | 18 | This paper introduces Agile Methods as the implementation of the principles of the lean production in software development. It discusses several agile practices that deal with requirements. These practices focus on a continuous interaction with the customer to address the requirements evolution over time, prioritize them, and deliver the most valuable functionalities first. | |
[SS05] is a general overview paper on requirements engineering for agile methods. The focus is not specifically on quality, but [SS05] mentions some criteria. | |||
Quality criteria: priority, non-functional requirements, independent, small/simple, customer language. |
Rights and permissions
About this article
Cite this article
Heck, P., Zaidman, A. A systematic literature review on quality criteria for agile requirements specifications. Software Qual J 26, 127–160 (2018). https://doi.org/10.1007/s11219-016-9336-4
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11219-016-9336-4