Application of machine learning techniques to the flexible assessment and improvement of requirements quality

Abstract

It is already common to compute quantitative metrics of requirements to assess their quality. However, the risk is to build assessment methods and tools that are both arbitrary and rigid in the parameterization and combination of metrics. Specifically, we show that a linear combination of metrics is insufficient to adequately compute a global measure of quality. In this work, we propose to develop a flexible method to assess and improve the quality of requirements that can be adapted to different contexts, projects, organizations, and quality standards, with a high degree of automation. The domain experts contribute with an initial set of requirements that they have classified according to their quality, and we extract their quality metrics. We then use machine learning techniques to emulate the implicit expert’s quality function. We provide also a procedure to suggest improvements in bad requirements. We compare the obtained rule-based classifiers with different machine learning algorithms, obtaining measurements of effectiveness around 85%. We show as well the appearance of the generated rules and how to interpret them. The method is tailorable to different contexts, different styles to write requirements, and different demands in quality. The whole process of inferring and applying the quality rules adapted to each organization is highly automated.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

References

  1. Aceituna, D., Walia, G., Do, H., & Lee, S. W. (2014). Model-based requirements verification method: Conclusions from two controlled experiments. Information and Software Technology, 56(3), 321–334.

    Article  Google Scholar 

  2. Alexander, I., Stevens, R. (2002). Writing better requirements. Addison-Wesley.

  3. Ali, R., Dalpiaz, F., & Giorgini, P. (2013). Reasoning with contextual requirements: detecting inconsistency and conflicts. Information and Software Technology, 55(1), 35–57.

    Article  Google Scholar 

  4. Belsis, P., Koutoumanos, A., & Sgouropoulou, C. (2014). PBURC: a patterns-based, unsupervised requirements clustering framework for distributed agile software development. Requirements Engineering, 19(2), 213–225.

    Article  Google Scholar 

  5. Berry, D.M., Bucchiarone, A., Gnesi, S., Lami, G., Trentanni, G. (2006). A new quality model for natural language requirements specifications. Proceedings of the 12th International Working Conference on Requirements Engineering: Foundation of Software Quality (REFSQ-06). Luxembourg, June 5-6 2006. Held in conjunction with CAiSE’06.

  6. Bishop, C.M. (2006). Pattern recognition and machine learning. Springer.

  7. Bøegh, J. (2008). A new standard for quality requirements. IEEE Software, 25(2), 57–63.

    Article  Google Scholar 

  8. Breiman, L. (1996). Bagging predictors. Machine Learning, 24(2), 123–140.

    MATH  Google Scholar 

  9. Brooks, F.P. (1987). No silver bullet. Essence and Accidents of Software Engineering. IEEE Computer 20(4):10-19. Reprinted in: F.P. Brooks. The Mytical Man-Month, Essays on Software Engineering. Addison-Wesley, 1995 (20th Anniversary Edition).

  10. Bucchiarone, A., Gnesi, S., Pierini, P. (2005). Quality analysis of NL requirements: an industrial case study. Proceedings of the 13th IEEE International Requirements Engineering Conference, pp. 390-394. Paris, France, August 29 – September 2, 2005.

  11. Cendrowska, J. (1987). PRISM: an algorithm for inducing modular rules. International Journal of Man-Machine Studies, 7(4), 349–370.

    MATH  Article  Google Scholar 

  12. Chalé-Góngora, H.G., Llorens, J., Gallego, E. (2017). Your wish, my command – speeding up projects in the transportation industry using ontologies.Proceedings of the 27th Annual INCOSE International Symposium (IS 2017), pp. 1070–1086. Adelaide, Australia, July 15-20, 2017.

  13. Chantree, F.J. (2006). Identifying nocuous ambiguity in natural language requirements. PhD Dissertation, The Open University, Faculty of Maths and Computing, UK.

  14. Clark, P., & Niblett, T. (1989). The CN2 induction algorithm. Machine Learning, 3(4), 261–283.

    Google Scholar 

  15. Cleland-Huang, J., Settimi, R., Zou, X., Solc, P. (2006). The detection and classification of non-functional requirements with application to early aspects. Proceedings of 14th IEEE International Requirements Engineering Conference (RE 2006), Minneapolis, MN, USA, September 11-15 2006, pp. 36–45.

  16. Cleland-Huang, J., Settimi, R., Zou, X., & Solc, P. (2007). Automated classification of non-functional requirements. Requirements Engineering, 12(2), 103–120.

    Article  Google Scholar 

  17. De Sousa, T. C., Almeida, J. R., Viana, S., & Pavón, J. (2010). Automatic analysis of requirements consistency with the B method. ACM SIGSOFT Software Engineering Notes, 35(2), 1–4.

    Article  Google Scholar 

  18. Dick, J., Wheatcraft, L., Long, D., Ryan, M., Llorens, J., Zinni, R., Svensson, C. (2017). Integrating requirement expressions with system models. International Council on Systems Engineering (INCOSE), Annual Systems Engineering Conference, ASEC 2017. University of Warwick, Coventry, UK., 21-22 November 2017.

  19. Dietterich, T. G. (1997). Machine learning research: four current directions. Artificial Intelligence Magazine, 18(4), 97–136.

    Google Scholar 

  20. Dietterich, T. G. (2000). An experimental comparison of three methods for constructing ensembles of decision trees: bagging, boosting, and randomization. Machine Learning, 40(2), 139–157.

    Article  Google Scholar 

  21. Dollmann, M., Geierhos, M. (2016). On- and off-topic classification and semantic annotation of user-generated software requirements. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (Austin, Texas, November 1-5 2016), pp. 1807–1816.

  22. Eito-Brun, R., & Amescua, A. (2017). Dealing with software process requirements complexity: an information access proposal based on semantic technologies. Requirements Engineering, 22(4), 527–542.

    Article  Google Scholar 

  23. European Space Agency. (1995). ESA PSS-05-03. Guide to the software requirements definition phase. ESA Board for Software Standardisation and Control (BSSC) (ftp://ftp.estec.esa.nl/pub/wm/anonymous/wme/bssc/PSS0503.pdf).

  24. Fabbrini, F., Fusani, M., Gnesi, S., Lami, G. (2001). The linguistic approach to the natural language requirements quality: benefit of the use of an automatic tool. Proceedings of the 26th Annual NASA Goddard Software Engineering Workshop, pp. 97–105.

  25. Fanmuy, G., Fraga, A., Llorens, J. (2011). Requirements verification in the industry. Proceedings of the Second International Conference on Complex Systems Design & Management CSDM 2011 (Paris, France, December 7–9, 2011), pp. 145–160.

  26. Frank, U. (2006). Towards a pluralistic conception of research methods in information systems research. ICB-Research Report No. 7. Institute for Computer Science and Business Information Systems, University Duisburg-Essen (http://www.icb.uni-due.de/fileadmin/ICB/research/research_reports/ICBReport07.pdf).

  27. Frank, E., Witten, I. H. (1998). Generating accurate rule sets without global optimization. Proceedings of the 15th International Conference on Machine Learning ICML-98 (Madison, WI, USA, July 24–27, 1998), pp. 144–151.

  28. Gallego, E., Chalé-Góngora, H.G., Llorens, J., Fuentes, J., Álvarez, J., Génova, G., Fraga, A. (2016). Requirements quality analysis: a successful case study in the industry. Proceedings of the Seventh International Conference on Complex Systems Design & Management, CSDM 2016, pp. 187–201. Paris, December 13–14, 2016.

  29. Génova, G., & González, M. R. (2016). Educational encounters of the third kind. Science and Engineering Ethics, 23(6), 1791–1800 December 2017.

    Article  Google Scholar 

  30. Génova, G., Llorens, J., Morato, J. (2012). Software engineering research: the need to strengthen and broaden the classical scientific method. In Manuel Mora, Ovsei Gelman, Annette L. Steenkamp and Mahesh Raisinghani (eds.), Research Methodologies, Innovations and Philosophies in Software Systems Engineering and Information Systems. IGI Global, 2012, pp. 106–125.

  31. Génova, G., Fuentes, J. M., Llorens, J., Hurtado, O., & Moreno, V. (2013). A framework to measure and improve the quality of textual requirements. Requirements Engineering, 18(1), 25–41.

    Article  Google Scholar 

  32. Ghaisas, S., Rose, P., Daneva, M., Sikkel, K., Wieringa, R. (2013). Generalizing by similarity: Lessons learnt from industrial case studies. Proceedings of the 1st International Workshop on Conducting Empirical Studies in Industry, CESI’13. Held at ICSE’13, 35th International Conference on Software Engineering, San Francisco, CA, USA, May 18–26, 2013, pp. 37–42.

  33. Gregory, S., & Terzakis, J. (2017). Viewpoint: effectiveness of focused mentoring to improve requirements engineering industrial practice. Requirements Engineering, 22(3), 413–417.

    Article  Google Scholar 

  34. Harman, M., & Jones, B. F. (2001). Search-based software engineering. Information and Software Technology, 43(14), 833–839.

    Article  Google Scholar 

  35. Harman, M., McMinn, P., De Souza, J., Yoo, S. (2012). Search based software engineering: techniques, taxonomy, tutorial. Empirical Software Engineering and Verification. International Summer Schools, LASER 2008-2010, Elba Island, Italy, Revised Tutorial Lectures. Springer Lecture Notes in Computer Science, 7007, pp. 1–59.

  36. Heck, P., Zaidman, A. (2018). A systematic literature review on quality criteria for agile requirements specifications. Software Quality Journal, published online 15 September 2016.

  37. Hevner, A. R., March, S. T., Park, J., & Ram, S. (2004). Design science in information systems research. MIS Quarterly, 28(1), 75–105.

    Article  Google Scholar 

  38. Hong, J., Mozetic, I. and Michalski, R.S. (1986). AQ15: Incremental learning of attribute-based descriptions from examples, the method and user’s guide. Report ISG 85–5, UIUCDCS-F-86-949, Department of Computer Science, University of Illinois at Urbana-Champaign.

  39. Hooks, I. (1993). Writing good requirements. Proceedings of the 3rd NCOSE International Symposium, 2, 1–12.

    Google Scholar 

  40. Huang, L., Ng, V., Persing, I., Chen, M., Li, Z., Geng, R., & Tian, J. (2015). AutoODC: automated generation of orthogonal defect classifications. Automated Software Engineering, 22(1), 3–46.

    Article  Google Scholar 

  41. Hussain, I., Ormandjieva, O., Kosseim, L. (2007). Automatic quality assessment of SRS text by means of a decision-tree-based text classifier. Proceedings of the Seventh International Conference on Quality Software (QSIC 2007), Portland, OR, 11–12 Oct. 2007, pp. 209–218.

  42. Hussain, I., Kosseim, L., Ormandjieva, O. (2008). Using linguistic knowledge to classify non-functional requirements in SRS documents. Proceedings of the International Conference on Application of Natural Language to Information Systems, NLDB 2008, London, UK, June 24–27 2008. Springer Lecture Notes in Computer Science, 5039, pp. 287–298.

  43. IEEE Std 830-1998. IEEE Recommended Practice for Software Requirements Specifications (http://ieeexplore.ieee.org/iel4/5841/15571/00720574.pdf).

  44. IEEE Computer Society. (2014). SWEBOK guide to the software engineering body of knowledge, Version 3.0. (http://www.computer.org/portal/web/swebok).

  45. International Council on Systems Engineering (INCOSE), Requirements Working Group. (2012). Guide for Writing Requirements. (http://www.incose.org/ProductsPublications/techpublications/GuideRequirements).

  46. ISO/IEC 25030:2007. Software engineeringsoftware product quality requirements and evaluation (SQuaRE)quality requirements (http://www.iso.org/iso/catalogue_detail.htm?csnumber=35755).

  47. James, L. (1999). Providing pragmatic advice on how good your requirements are - the precept ‘requirements councillor’ utility. Proceedings of the 9th INCOSE International Symposium, Brighton, England, 6–11 June 1999, pp. 1427–1430.

  48. Jani, H., Islam, A. (2012). A framework of software requirements quality analysis system using case-based reasoning and neural network. (2012). Proceedings of the 6th international conference on new trends in information science and service science and data mining (ISSDM), Taipei, 23–25 Oct. 2012, pp. 152–157.

  49. Kasser, J.E., Scott, W., Tran, X.L., Nesterov, S. (2006). A proposed research programme for determining a metric for a good requirement. The Conference on Systems Engineering Research, Los Angeles, California, USA, 2006.

  50. Kiyavitskaya, N., Zeni, N., Mich, L., & Berry, D. M. (2008). Requirements for tools for ambiguity identification and measurement in natural language requirements specifications. Requirements Engineering, 13(3), 207–239.

    Article  Google Scholar 

  51. Ko, Y., Park, S., Seo, J., & Choi, S. (2007). Using classification techniques for informal requirements in the requirements analysis-supporting system. Information and Software Technology, 49(11–12), 1128–1140.

    Article  Google Scholar 

  52. Kohavi, R (1995). A study of cross-validation and bootstrap for accuracy estimation and model selection. Proceedings of the Fourteenth International Joint Conference on Artificial Intelligence IJCAI-95 (Montreal, Quebec, Canada, August 20–25, 1995), vol. 2, pp. 1137–1143.

  53. Loucopoulos, P., Karakostas, V. (1985). Systems requirements engineering. McGraw-Hill.

  54. Lucassen, G., Dalpiaz, F., Van Der Werf, J. M., & Brinkkemper, S. (2016). Improving agile requirements: the quality user story framework and tool. Requirements Engineering, 21(3), 383–403.

    Article  Google Scholar 

  55. Magee, S., & Tripp, L. L. (1997). Guide to software engineering standards and specifications. Boston: Artech House.

    MATH  Google Scholar 

  56. Major, J. A., & Mangano, J. J. (1995). Selecting among rules induced from a hurricane database. Journal of Intelligent Information Systems, 4(1), 39–52.

    Article  Google Scholar 

  57. Marsick, V.J., Volpe, M. (1999). (Eds.). Informal learning on the job. Berrett-Koehler Publishers.

  58. Merton, R. (1968). On sociological theories of the middle range. In: Social Theory and Social Structure, enlarged edition, The Free Press, 1968, pp.39–72.

  59. Mich, L., Franch, M., & Inverardi, P. L. N. (2004). Market research for requirements analysis using linguistic tools. Requirements Engineering, 9(1), 40–56.

    Article  Google Scholar 

  60. Moreno, V., Génova, G., Parra, E., Fraga, A. (2016). Metrics obtained with the RQA tool from a set of 1035 requirements provided by INCOSE Requirements Working Group. July 2016 (available at https://www.dropbox.com/s/ri38j4mmwjkl477/INCOSE-RQA-20-metrics-1035-requirements.pdf).

  61. Otero, C.E., Dell, E., Qureshi, A., Otero, L.D. (2010). A quality-based requirement prioritization framework using binary inputs. Proceedings of the 4th Asia International Conference on Mathematical/Analytical Modelling and Computer Simulation, Kota Kinabalu, Malaysia, 26–28 May 2010, pp. 187–192.

  62. Ott, D. (2013). Automatic requirement categorization of large natural language specifications at Mercedes-Benz for review improvements. Requirements Engineering: Foundation for Software Quality. Proceedings of the 19th International Working Conference, REFSQ 2013, Essen, Germany, April 8-11, 2013. Springer Lecture Notes in Computer Science, 7830, pp. 50–64.

  63. Parra, E., Dimou, C., Llorens, J., Moreno, V., & Fraga, A. (2015). A methodology for the classification of quality of requirements using machine learning techniques. Information and Software Technology, 67, 180–195.

    Article  Google Scholar 

  64. Popescu, D., Rugaber, S., Medvidovic, N., Berry, D.M. (2008). Reducing ambiguities in requirements specifications via automatically created object-oriented models. Proceedings of the 14th Monterey Workshop on Requirements Analysis. Monterey, CA, USA, September 10-13 2007. Springer Lecture Notes in Computer Science, 5320, pp. 103-124.

  65. Quinlan, J. R. (1986). Induction of decision trees (ID3 algorithm). Machine Learning, 1(1), 81–106.

    Google Scholar 

  66. Quinlan, J. R. (1993). C4.5: programs for machine learning. CA: Morgan Kaufmann.

    Google Scholar 

  67. Rashwan, A. (2015). Automated quality assurance of non-functional requirements for testability. Masters thesis, Concordia University, Montréal, Québec, Canada, April 2015.

  68. Robledano, J., Moreno, V., & Pereira, J. M. (2016). Aproximación experimental al uso de métricas objetivas para la estimación de calidad cromática en la digitalización de patrimonio documental gráfico. Revista Española de Documentación Científica, 39(2), e128.

    Article  Google Scholar 

  69. Rosenberg, L.H., Linda, H. (2001). Generating high quality requirements. Proceedings of the AIAA Space 2001 Conference and Exposition, AIAA Paper 2001-4524. American Institute of Aeronautics and Astronautics, Albuquerque, NM, August 28-30, 2001.

  70. Russell, S. and Norvig, P. (2003). Artificial intelligence: a modern approach (2nd ed.). Prentice Hall.

  71. Sardinha, A., Chitchyan, R., Weston, N., Greenwood, P., & Rashid, A. (2013). EA-analyzer: automating conflict detection in a large set of textual aspect-oriented requirements. Automated Software Engineering, 20(1), 111–135.

    Article  Google Scholar 

  72. Schapire, R. E. (1990). The strength of weak learnability. Machine Learning, 5(2), 197–227.

    Google Scholar 

  73. Siahaan, D., & Umami, I. (2011). Natural language processing for detecting forward reference in a document. IPTEK, The Journal for Technology and Science, 22(4), 138–142.

    Google Scholar 

  74. Terzakis, J., Gregory, S. (2016). RAMP: requirements authors mentoring program. Proceedings of the 24th International Requirements Engineering Conference (RE 2016), Beijing, China, September 12-16 2016, pp. 323–328.

  75. Thakurta, R. (2013). A framework for prioritization of quality requirements for inclusion in a software project. Software Quality Journal, 21(4), 573–597.

    Article  Google Scholar 

  76. The Reuse Company. (2016). RQA requirements quality analyzer (http://www.reusecompany.com/requirements-quality-analyzer).

  77. The Standish Group. (2015). Chaos report (http://www.standishgroup.com/).

  78. Thitisathienkul, P., Prompoon, N. (2015). Quality assessment method for software requirements specifications based on document characteristics and its structure. Proceedings of the Second International Conference on Trustworthy Systems and Their Applications (TSA 2015), Hualien, Taiwan, July 8–9 2015, pp. 51–60.

  79. Turk, W. (2006). Writing requirements for engineers. IET Engineering Management, 16(3), 20–23.

    Google Scholar 

  80. Wang, Y., Gutiérrez, I.L.M., Winbladh, K., Fang, H. (2013). Automatic detection of ambiguous terminology for software requirements. In Natural Language Processing and Information Systems, Proceedings of the 18th International Conference on Applications of Natural Language to Information Systems, NLDB 2013, Salford, UK, June 19–21, 2013. Springer Lecture Notes in Computer Science, 7934, pp. 25–37.

  81. Weiss, S.M., Indurkhya, N. (1998). Predictive data mining: a practical guide. Morgan Kaufmann.

  82. Wieringa, R. (2014). Design science methodology for information systems and software engineering. Springer.

  83. Wieringa, R., & Daneva, M. (2015). Six strategies for generalizing software engineering theories. Science of Computer Programming, 101, 136–152.

    Article  Google Scholar 

  84. Wilson, W.M., Rosenberg, L.H., Hyatt, L.E. (1997). Automated analysis of requirement specifications. Proceedings of the 19th International Conference on Software Engineering-ICSE’97, May 17–23, 1997, Boston, Massachusetts, USA, pp. 161–171.

  85. Witten, I.H. and Frank, E. (2000). Data mining: practical machine learning tools and techniques with Java implementations. Morgan Kaufmann.

  86. Wolpert, D. H. (1992). Stacked generalization. Neural Networks, 5(2), 241–259.

    Article  Google Scholar 

  87. Zhang, Y., Harman, M., Mansouri, S. (2007). The multi-objective next release problem. Proceedings of the 9th Annual Conference on Genetic and Evolutionary Computation (GECCO ‘07), London July 7–11, 2007, pp. 1129–1137.

Download references

Acknowledgments

This research would not have been possible without the support and help of Gauthier Fanmuy, expert in Requirements Engineering and Model-Based Systems Engineering, member of INCOSE (International Council on Systems Engineering) and AFIS (Association Française d’Ingénierie Système).

Funding

This research has received funding from the CRYSTAL project–Critical System Engineering Acceleration (European Union’s Seventh Framework Program FP7/2007-2013, ARTEMIS Joint Undertaking grant agreement no 332830); and from the AMASS project–Architecture-driven, Multi-concern and Seamless Assurance and Certification of Cyber-Physical Systems (H2020-ECSEL grant agreement no 692474; Spain’s MINECO ref. PCIN-2015-262).

Author information

Affiliations

Authors

Corresponding author

Correspondence to Gonzalo Génova.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendix

Appendix

Table 3 Description of quality metrics. For a more complete description and justification of the metrics, see our previous works (Génova et al. 2013; Parra et al. 2015).

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Moreno, V., Génova, G., Parra, E. et al. Application of machine learning techniques to the flexible assessment and improvement of requirements quality. Software Qual J 28, 1645–1674 (2020). https://doi.org/10.1007/s11219-020-09511-4

Download citation

Keywords

  • Requirements quality
  • Machine learning
  • Automatic classification
  • Automatic improvement
  • Experts’ judgment
  • Flexible assessment