Abstract
[Context and motivation] Incompleteness in natural-language requirements is a challenging problem. [Question/problem] A common technique for detecting incompleteness in requirements is checking the requirements against external sources. With the emergence of language models such as BERT, an interesting question is whether language models are useful external sources for finding potential incompleteness in requirements. [Principal ideas/results] We mask words in requirements and have BERT’s masked language model (MLM) generate contextualized predictions for filling the masked slots. We simulate incompleteness by withholding content from requirements and measure BERT’s ability to predict terminology that is present in the withheld content but absent in the content disclosed to BERT. [Contribution] BERT can be configured to generate multiple predictions per mask. Our first contribution is to determine how many predictions per mask is an optimal trade-off between effectively discovering omissions in requirements and the level of noise in the predictions. Our second contribution is devising a machine learning-based filter that post-processes predictions made by BERT to further reduce noise. We empirically evaluate our solution over 40 requirements specifications drawn from the PURE dataset [1]. Our results indicate that: (1) predictions made by BERT are highly effective at pinpointing terminology that is missing from requirements, and (2) our filter can substantially reduce noise from the predictions, thus making BERT a more compelling aid for improving completeness in requirements.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Ferrari, A., Spagnolo, G.O., Gnesi, S.: PURE: a dataset of public requirements documents. In: RE (2017)
Zowghi, D., Gervasi, V.: The three Cs of requirements: consistency, completeness, and correctness. In: REFSQ (2003)
Zowghi, D., Gervasi, V.: On the interplay between consistency, completeness, and correctness in requirements evolution. IST 45(14), 993–1009 (2003)
Arora, C., Sabetzadeh, M., Briand, L.C.: An empirical study on the potential usefulness of domain models for completeness checking of requirements. Empir. Softw. Eng. 24(4), 2509–2539 (2019). https://doi.org/10.1007/s10664-019-09693-x
Ferrari, A., dell’Orletta, F., Spagnolo, G.O., Gnesi, S.: Measuring and improving the completeness of natural language requirements. In: Salinesi, C., van de Weerd, I. (eds.) REFSQ 2014. LNCS, vol. 8396, pp. 23–38. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-05843-6_3
Dalpiaz, F., van der Schalk, I., Lucassen, G.: Pinpointing ambiguity and incompleteness in requirements engineering via information visualization and NLP. In: Kamsties, E., Horkoff, J., Dalpiaz, F. (eds.) REFSQ 2018. LNCS, vol. 10753, pp. 119–135. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-77243-1_8
Devlin, J., Chang, M.-W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. In: NAACL-HLT (2019)
Luitel, D., Hassani, S., Sabetzadeh, M.: Replication package (2023). https://doi.org/10.6084/m9.figshare.22041341
Jurafsky, D., Martin, J.: Speech and Language Processing, 2nd edn. Prentice Hall, Upper Saddle River (2009)
Hey, T., Keim, J., Koziolek, A., Tichy, W.F.: NoRBERT: transfer learning for requirements classification. In: RE (2020)
Ezzini, S., Abualhaija, S., Arora, C., Sabetzadeh, M.: Automated handling of anaphoric ambiguity in requirements: a multi-solution study. In: ICSE (2022)
Mikolov, T., Yih, W., Zweig, G.: Linguistic regularities in continuous space word representations. In: NAACL-HLT (2013)
Pennington, J., Socher, R., Manning, C.: GloVe: global vectors for word representation. In: EMNLP (2014)
Witten, I.H., Frank, E., Hall, M.A.: Data Mining: Practical Machine Learning Tools and Techniques, 4th edn. Morgan Kaufmann, Boston (2017)
Berry, D.M., Cleland-Huang, J., Ferrari, A., Maalej, W., Mylopoulos, J., Zowghi, D.: Panel: context-dependent evaluation of tools for NL RE tasks: recall vs. precision, and beyond. In: RE (2017)
Ezzini, S., Abualhaija, S., Arora, C., Sabetzadeh, M., Briand, L.: Using domain-specific corpora for improved handling of ambiguity in requirements. In: ICSE (2021)
Cui, G., Lu, Q., Li, W., Chen, Y.R.: Corpus exploitation from Wikipedia for ontology construction. In: LREC (2008)
Ferrari, A., Donati, B., Gnesi, S.: Detecting domain-specific ambiguities: an NLP approach based on Wikipedia crawling and word embeddings. In: AIRE (2017)
Ezzini, S., Abualhaija, S., Sabetzadeh, M.: WikiDoMiner: wikipedia domain-specific miner. In: ESEC/FSE (2022)
Daniel, M., Berry, E.K., Krieger, M.: From contract drafting to software specification: linguistic sources of ambiguity, a handbook (2003)
Arora, C., Sabetzadeh, M., Briand, L., Zimmer, F.: Automated checking of conformance to requirements templates using natural language processing. IEEE TSE 41(10), 944–968 (2015)
Arora, C., Sabetzadeh, M., Briand, L., Zimmer, F.: Automated extraction and clustering of requirements glossary terms. IEEE TSE 43(10), 918–945 (2017)
Witten, I.H., Frank, E., Hall, M.A., Pal, C.J.: The WEKA Workbench: Online Appendix for “Data Mining: Practical Machine Learning Tools and Techniques", 4th edn. Morgan Kaufmann Publishers Inc., Boston (2016)
Manning, C., Raghavan, P., Schütze, H.: Introduction to Information Retrieval. In: Syngress (2008)
Bergstra, J., Bengio, Y.: Random search for hyper-parameter optimization. JMLR 13(2), 1–25 (2012)
Capon, J.A.: Elementary Statistics for the Social Sciences: Study Guide. In: Wadsworth (1991)
Vargha, A., Delaney, H.: A critique and improvement of the CL common language effect size statistics of McGraw and Wong. J. Educ. Behav. Stat. 25(2), 101–132 (2000)
Bhatia, J., Breaux, T.: Semantic incompleteness in privacy policy goals. In: RE (2018)
Cejas, O.A., Abualhaija, S., Torre, D., Sabetzadeh, M., Briand, L.: AI-enabled automation for completeness checking of privacy policies. IEEE TSE 48(11), 4647–4674 (2022)
Shen, Y., Breaux, T.: Domain model extraction from user-authored scenarios and word embeddings. In: AIRE (2022)
Acknowledgements
This work was funded by the Natural Sciences and Engineering Research Council of Canada (NSERC) under the Discovery and Discovery Accelerator programs. We are grateful to Shiva Nejati, Sallam Abualhaija and Jia Li for helpful discussions. We thank the anonymous reviewers of REFSQ 2023 for their constructive comments.
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Luitel, D., Hassani, S., Sabetzadeh, M. (2023). Using Language Models for Enhancing the Completeness of Natural-Language Requirements. In: Ferrari, A., Penzenstadler, B. (eds) Requirements Engineering: Foundation for Software Quality. REFSQ 2023. Lecture Notes in Computer Science, vol 13975. Springer, Cham. https://doi.org/10.1007/978-3-031-29786-1_7
Download citation
DOI: https://doi.org/10.1007/978-3-031-29786-1_7
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-29785-4
Online ISBN: 978-3-031-29786-1
eBook Packages: Computer ScienceComputer Science (R0)