It is already common to compute quantitative metrics of requirements to assess their quality. However, the risk is to build assessment methods and tools that are both arbitrary and rigid in the parameterization and combination of metrics. Specifically, we show that a linear combination of metrics is insufficient to adequately compute a global measure of quality. In this work, we propose to develop a flexible method to assess and improve the quality of requirements that can be adapted to different contexts, projects, organizations, and quality standards, with a high degree of automation. The domain experts contribute with an initial set of requirements that they have classified according to their quality, and we extract their quality metrics. We then use machine learning techniques to emulate the implicit expert’s quality function. We provide also a procedure to suggest improvements in bad requirements. We compare the obtained rule-based classifiers with different machine learning algorithms, obtaining measurements of effectiveness around 85%. We show as well the appearance of the generated rules and how to interpret them. The method is tailorable to different contexts, different styles to write requirements, and different demands in quality. The whole process of inferring and applying the quality rules adapted to each organization is highly automated.
This is a preview of subscription content, access via your institution.
Buy single article
Instant access to the full article PDF.
Tax calculation will be finalised during checkout.
Subscribe to journal
Immediate online access to all issues from 2019. Subscription will auto renew annually.
Tax calculation will be finalised during checkout.
Aceituna, D., Walia, G., Do, H., & Lee, S. W. (2014). Model-based requirements verification method: Conclusions from two controlled experiments. Information and Software Technology, 56(3), 321–334.
Alexander, I., Stevens, R. (2002). Writing better requirements. Addison-Wesley.
Ali, R., Dalpiaz, F., & Giorgini, P. (2013). Reasoning with contextual requirements: detecting inconsistency and conflicts. Information and Software Technology, 55(1), 35–57.
Belsis, P., Koutoumanos, A., & Sgouropoulou, C. (2014). PBURC: a patterns-based, unsupervised requirements clustering framework for distributed agile software development. Requirements Engineering, 19(2), 213–225.
Berry, D.M., Bucchiarone, A., Gnesi, S., Lami, G., Trentanni, G. (2006). A new quality model for natural language requirements specifications. Proceedings of the 12th International Working Conference on Requirements Engineering: Foundation of Software Quality (REFSQ-06). Luxembourg, June 5-6 2006. Held in conjunction with CAiSE’06.
Bishop, C.M. (2006). Pattern recognition and machine learning. Springer.
Bøegh, J. (2008). A new standard for quality requirements. IEEE Software, 25(2), 57–63.
Breiman, L. (1996). Bagging predictors. Machine Learning, 24(2), 123–140.
Brooks, F.P. (1987). No silver bullet. Essence and Accidents of Software Engineering. IEEE Computer 20(4):10-19. Reprinted in: F.P. Brooks. The Mytical Man-Month, Essays on Software Engineering. Addison-Wesley, 1995 (20th Anniversary Edition).
Bucchiarone, A., Gnesi, S., Pierini, P. (2005). Quality analysis of NL requirements: an industrial case study. Proceedings of the 13th IEEE International Requirements Engineering Conference, pp. 390-394. Paris, France, August 29 – September 2, 2005.
Cendrowska, J. (1987). PRISM: an algorithm for inducing modular rules. International Journal of Man-Machine Studies, 7(4), 349–370.
Chalé-Góngora, H.G., Llorens, J., Gallego, E. (2017). Your wish, my command – speeding up projects in the transportation industry using ontologies.Proceedings of the 27th Annual INCOSE International Symposium (IS 2017), pp. 1070–1086. Adelaide, Australia, July 15-20, 2017.
Chantree, F.J. (2006). Identifying nocuous ambiguity in natural language requirements. PhD Dissertation, The Open University, Faculty of Maths and Computing, UK.
Clark, P., & Niblett, T. (1989). The CN2 induction algorithm. Machine Learning, 3(4), 261–283.
Cleland-Huang, J., Settimi, R., Zou, X., Solc, P. (2006). The detection and classification of non-functional requirements with application to early aspects. Proceedings of 14th IEEE International Requirements Engineering Conference (RE 2006), Minneapolis, MN, USA, September 11-15 2006, pp. 36–45.
Cleland-Huang, J., Settimi, R., Zou, X., & Solc, P. (2007). Automated classification of non-functional requirements. Requirements Engineering, 12(2), 103–120.
De Sousa, T. C., Almeida, J. R., Viana, S., & Pavón, J. (2010). Automatic analysis of requirements consistency with the B method. ACM SIGSOFT Software Engineering Notes, 35(2), 1–4.
Dick, J., Wheatcraft, L., Long, D., Ryan, M., Llorens, J., Zinni, R., Svensson, C. (2017). Integrating requirement expressions with system models. International Council on Systems Engineering (INCOSE), Annual Systems Engineering Conference, ASEC 2017. University of Warwick, Coventry, UK., 21-22 November 2017.
Dietterich, T. G. (1997). Machine learning research: four current directions. Artificial Intelligence Magazine, 18(4), 97–136.
Dietterich, T. G. (2000). An experimental comparison of three methods for constructing ensembles of decision trees: bagging, boosting, and randomization. Machine Learning, 40(2), 139–157.
Dollmann, M., Geierhos, M. (2016). On- and off-topic classification and semantic annotation of user-generated software requirements. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (Austin, Texas, November 1-5 2016), pp. 1807–1816.
Eito-Brun, R., & Amescua, A. (2017). Dealing with software process requirements complexity: an information access proposal based on semantic technologies. Requirements Engineering, 22(4), 527–542.
European Space Agency. (1995). ESA PSS-05-03. Guide to the software requirements definition phase. ESA Board for Software Standardisation and Control (BSSC) (ftp://ftp.estec.esa.nl/pub/wm/anonymous/wme/bssc/PSS0503.pdf).
Fabbrini, F., Fusani, M., Gnesi, S., Lami, G. (2001). The linguistic approach to the natural language requirements quality: benefit of the use of an automatic tool. Proceedings of the 26th Annual NASA Goddard Software Engineering Workshop, pp. 97–105.
Fanmuy, G., Fraga, A., Llorens, J. (2011). Requirements verification in the industry. Proceedings of the Second International Conference on Complex Systems Design & Management CSDM 2011 (Paris, France, December 7–9, 2011), pp. 145–160.
Frank, U. (2006). Towards a pluralistic conception of research methods in information systems research. ICB-Research Report No. 7. Institute for Computer Science and Business Information Systems, University Duisburg-Essen (http://www.icb.uni-due.de/fileadmin/ICB/research/research_reports/ICBReport07.pdf).
Frank, E., Witten, I. H. (1998). Generating accurate rule sets without global optimization. Proceedings of the 15th International Conference on Machine Learning ICML-98 (Madison, WI, USA, July 24–27, 1998), pp. 144–151.
Gallego, E., Chalé-Góngora, H.G., Llorens, J., Fuentes, J., Álvarez, J., Génova, G., Fraga, A. (2016). Requirements quality analysis: a successful case study in the industry. Proceedings of the Seventh International Conference on Complex Systems Design & Management, CSDM 2016, pp. 187–201. Paris, December 13–14, 2016.
Génova, G., & González, M. R. (2016). Educational encounters of the third kind. Science and Engineering Ethics, 23(6), 1791–1800 December 2017.
Génova, G., Llorens, J., Morato, J. (2012). Software engineering research: the need to strengthen and broaden the classical scientific method. In Manuel Mora, Ovsei Gelman, Annette L. Steenkamp and Mahesh Raisinghani (eds.), Research Methodologies, Innovations and Philosophies in Software Systems Engineering and Information Systems. IGI Global, 2012, pp. 106–125.
Génova, G., Fuentes, J. M., Llorens, J., Hurtado, O., & Moreno, V. (2013). A framework to measure and improve the quality of textual requirements. Requirements Engineering, 18(1), 25–41.
Ghaisas, S., Rose, P., Daneva, M., Sikkel, K., Wieringa, R. (2013). Generalizing by similarity: Lessons learnt from industrial case studies. Proceedings of the 1st International Workshop on Conducting Empirical Studies in Industry, CESI’13. Held at ICSE’13, 35th International Conference on Software Engineering, San Francisco, CA, USA, May 18–26, 2013, pp. 37–42.
Gregory, S., & Terzakis, J. (2017). Viewpoint: effectiveness of focused mentoring to improve requirements engineering industrial practice. Requirements Engineering, 22(3), 413–417.
Harman, M., & Jones, B. F. (2001). Search-based software engineering. Information and Software Technology, 43(14), 833–839.
Harman, M., McMinn, P., De Souza, J., Yoo, S. (2012). Search based software engineering: techniques, taxonomy, tutorial. Empirical Software Engineering and Verification. International Summer Schools, LASER 2008-2010, Elba Island, Italy, Revised Tutorial Lectures. Springer Lecture Notes in Computer Science, 7007, pp. 1–59.
Heck, P., Zaidman, A. (2018). A systematic literature review on quality criteria for agile requirements specifications. Software Quality Journal, published online 15 September 2016.
Hevner, A. R., March, S. T., Park, J., & Ram, S. (2004). Design science in information systems research. MIS Quarterly, 28(1), 75–105.
Hong, J., Mozetic, I. and Michalski, R.S. (1986). AQ15: Incremental learning of attribute-based descriptions from examples, the method and user’s guide. Report ISG 85–5, UIUCDCS-F-86-949, Department of Computer Science, University of Illinois at Urbana-Champaign.
Hooks, I. (1993). Writing good requirements. Proceedings of the 3rd NCOSE International Symposium, 2, 1–12.
Huang, L., Ng, V., Persing, I., Chen, M., Li, Z., Geng, R., & Tian, J. (2015). AutoODC: automated generation of orthogonal defect classifications. Automated Software Engineering, 22(1), 3–46.
Hussain, I., Ormandjieva, O., Kosseim, L. (2007). Automatic quality assessment of SRS text by means of a decision-tree-based text classifier. Proceedings of the Seventh International Conference on Quality Software (QSIC 2007), Portland, OR, 11–12 Oct. 2007, pp. 209–218.
Hussain, I., Kosseim, L., Ormandjieva, O. (2008). Using linguistic knowledge to classify non-functional requirements in SRS documents. Proceedings of the International Conference on Application of Natural Language to Information Systems, NLDB 2008, London, UK, June 24–27 2008. Springer Lecture Notes in Computer Science, 5039, pp. 287–298.
IEEE Std 830-1998. IEEE Recommended Practice for Software Requirements Specifications (http://ieeexplore.ieee.org/iel4/5841/15571/00720574.pdf).
IEEE Computer Society. (2014). SWEBOK guide to the software engineering body of knowledge, Version 3.0. (http://www.computer.org/portal/web/swebok).
International Council on Systems Engineering (INCOSE), Requirements Working Group. (2012). Guide for Writing Requirements. (http://www.incose.org/ProductsPublications/techpublications/GuideRequirements).
ISO/IEC 25030:2007. Software engineering—software product quality requirements and evaluation (SQuaRE)—quality requirements (http://www.iso.org/iso/catalogue_detail.htm?csnumber=35755).
James, L. (1999). Providing pragmatic advice on how good your requirements are - the precept ‘requirements councillor’ utility. Proceedings of the 9th INCOSE International Symposium, Brighton, England, 6–11 June 1999, pp. 1427–1430.
Jani, H., Islam, A. (2012). A framework of software requirements quality analysis system using case-based reasoning and neural network. (2012). Proceedings of the 6th international conference on new trends in information science and service science and data mining (ISSDM), Taipei, 23–25 Oct. 2012, pp. 152–157.
Kasser, J.E., Scott, W., Tran, X.L., Nesterov, S. (2006). A proposed research programme for determining a metric for a good requirement. The Conference on Systems Engineering Research, Los Angeles, California, USA, 2006.
Kiyavitskaya, N., Zeni, N., Mich, L., & Berry, D. M. (2008). Requirements for tools for ambiguity identification and measurement in natural language requirements specifications. Requirements Engineering, 13(3), 207–239.
Ko, Y., Park, S., Seo, J., & Choi, S. (2007). Using classification techniques for informal requirements in the requirements analysis-supporting system. Information and Software Technology, 49(11–12), 1128–1140.
Kohavi, R (1995). A study of cross-validation and bootstrap for accuracy estimation and model selection. Proceedings of the Fourteenth International Joint Conference on Artificial Intelligence IJCAI-95 (Montreal, Quebec, Canada, August 20–25, 1995), vol. 2, pp. 1137–1143.
Loucopoulos, P., Karakostas, V. (1985). Systems requirements engineering. McGraw-Hill.
Lucassen, G., Dalpiaz, F., Van Der Werf, J. M., & Brinkkemper, S. (2016). Improving agile requirements: the quality user story framework and tool. Requirements Engineering, 21(3), 383–403.
Magee, S., & Tripp, L. L. (1997). Guide to software engineering standards and specifications. Boston: Artech House.
Major, J. A., & Mangano, J. J. (1995). Selecting among rules induced from a hurricane database. Journal of Intelligent Information Systems, 4(1), 39–52.
Marsick, V.J., Volpe, M. (1999). (Eds.). Informal learning on the job. Berrett-Koehler Publishers.
Merton, R. (1968). On sociological theories of the middle range. In: Social Theory and Social Structure, enlarged edition, The Free Press, 1968, pp.39–72.
Mich, L., Franch, M., & Inverardi, P. L. N. (2004). Market research for requirements analysis using linguistic tools. Requirements Engineering, 9(1), 40–56.
Moreno, V., Génova, G., Parra, E., Fraga, A. (2016). Metrics obtained with the RQA tool from a set of 1035 requirements provided by INCOSE Requirements Working Group. July 2016 (available at https://www.dropbox.com/s/ri38j4mmwjkl477/INCOSE-RQA-20-metrics-1035-requirements.pdf).
Otero, C.E., Dell, E., Qureshi, A., Otero, L.D. (2010). A quality-based requirement prioritization framework using binary inputs. Proceedings of the 4th Asia International Conference on Mathematical/Analytical Modelling and Computer Simulation, Kota Kinabalu, Malaysia, 26–28 May 2010, pp. 187–192.
Ott, D. (2013). Automatic requirement categorization of large natural language specifications at Mercedes-Benz for review improvements. Requirements Engineering: Foundation for Software Quality. Proceedings of the 19th International Working Conference, REFSQ 2013, Essen, Germany, April 8-11, 2013. Springer Lecture Notes in Computer Science, 7830, pp. 50–64.
Parra, E., Dimou, C., Llorens, J., Moreno, V., & Fraga, A. (2015). A methodology for the classification of quality of requirements using machine learning techniques. Information and Software Technology, 67, 180–195.
Popescu, D., Rugaber, S., Medvidovic, N., Berry, D.M. (2008). Reducing ambiguities in requirements specifications via automatically created object-oriented models. Proceedings of the 14th Monterey Workshop on Requirements Analysis. Monterey, CA, USA, September 10-13 2007. Springer Lecture Notes in Computer Science, 5320, pp. 103-124.
Quinlan, J. R. (1986). Induction of decision trees (ID3 algorithm). Machine Learning, 1(1), 81–106.
Quinlan, J. R. (1993). C4.5: programs for machine learning. CA: Morgan Kaufmann.
Rashwan, A. (2015). Automated quality assurance of non-functional requirements for testability. Masters thesis, Concordia University, Montréal, Québec, Canada, April 2015.
Robledano, J., Moreno, V., & Pereira, J. M. (2016). Aproximación experimental al uso de métricas objetivas para la estimación de calidad cromática en la digitalización de patrimonio documental gráfico. Revista Española de Documentación Científica, 39(2), e128.
Rosenberg, L.H., Linda, H. (2001). Generating high quality requirements. Proceedings of the AIAA Space 2001 Conference and Exposition, AIAA Paper 2001-4524. American Institute of Aeronautics and Astronautics, Albuquerque, NM, August 28-30, 2001.
Russell, S. and Norvig, P. (2003). Artificial intelligence: a modern approach (2nd ed.). Prentice Hall.
Sardinha, A., Chitchyan, R., Weston, N., Greenwood, P., & Rashid, A. (2013). EA-analyzer: automating conflict detection in a large set of textual aspect-oriented requirements. Automated Software Engineering, 20(1), 111–135.
Schapire, R. E. (1990). The strength of weak learnability. Machine Learning, 5(2), 197–227.
Siahaan, D., & Umami, I. (2011). Natural language processing for detecting forward reference in a document. IPTEK, The Journal for Technology and Science, 22(4), 138–142.
Terzakis, J., Gregory, S. (2016). RAMP: requirements authors mentoring program. Proceedings of the 24th International Requirements Engineering Conference (RE 2016), Beijing, China, September 12-16 2016, pp. 323–328.
Thakurta, R. (2013). A framework for prioritization of quality requirements for inclusion in a software project. Software Quality Journal, 21(4), 573–597.
The Reuse Company. (2016). RQA requirements quality analyzer (http://www.reusecompany.com/requirements-quality-analyzer).
The Standish Group. (2015). Chaos report (http://www.standishgroup.com/).
Thitisathienkul, P., Prompoon, N. (2015). Quality assessment method for software requirements specifications based on document characteristics and its structure. Proceedings of the Second International Conference on Trustworthy Systems and Their Applications (TSA 2015), Hualien, Taiwan, July 8–9 2015, pp. 51–60.
Turk, W. (2006). Writing requirements for engineers. IET Engineering Management, 16(3), 20–23.
Wang, Y., Gutiérrez, I.L.M., Winbladh, K., Fang, H. (2013). Automatic detection of ambiguous terminology for software requirements. In Natural Language Processing and Information Systems, Proceedings of the 18th International Conference on Applications of Natural Language to Information Systems, NLDB 2013, Salford, UK, June 19–21, 2013. Springer Lecture Notes in Computer Science, 7934, pp. 25–37.
Weiss, S.M., Indurkhya, N. (1998). Predictive data mining: a practical guide. Morgan Kaufmann.
Wieringa, R. (2014). Design science methodology for information systems and software engineering. Springer.
Wieringa, R., & Daneva, M. (2015). Six strategies for generalizing software engineering theories. Science of Computer Programming, 101, 136–152.
Wilson, W.M., Rosenberg, L.H., Hyatt, L.E. (1997). Automated analysis of requirement specifications. Proceedings of the 19th International Conference on Software Engineering-ICSE’97, May 17–23, 1997, Boston, Massachusetts, USA, pp. 161–171.
Witten, I.H. and Frank, E. (2000). Data mining: practical machine learning tools and techniques with Java implementations. Morgan Kaufmann.
Wolpert, D. H. (1992). Stacked generalization. Neural Networks, 5(2), 241–259.
Zhang, Y., Harman, M., Mansouri, S. (2007). The multi-objective next release problem. Proceedings of the 9th Annual Conference on Genetic and Evolutionary Computation (GECCO ‘07), London July 7–11, 2007, pp. 1129–1137.
This research would not have been possible without the support and help of Gauthier Fanmuy, expert in Requirements Engineering and Model-Based Systems Engineering, member of INCOSE (International Council on Systems Engineering) and AFIS (Association Française d’Ingénierie Système).
This research has received funding from the CRYSTAL project–Critical System Engineering Acceleration (European Union’s Seventh Framework Program FP7/2007-2013, ARTEMIS Joint Undertaking grant agreement no 332830); and from the AMASS project–Architecture-driven, Multi-concern and Seamless Assurance and Certification of Cyber-Physical Systems (H2020-ECSEL grant agreement no 692474; Spain’s MINECO ref. PCIN-2015-262).
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Moreno, V., Génova, G., Parra, E. et al. Application of machine learning techniques to the flexible assessment and improvement of requirements quality. Software Qual J 28, 1645–1674 (2020). https://doi.org/10.1007/s11219-020-09511-4
- Requirements quality
- Machine learning
- Automatic classification
- Automatic improvement
- Experts’ judgment
- Flexible assessment