Abstract
In this paper, we address the question of whether general-purpose LLM-based tools may be useful for detecting requirements variability in Natural Language (NL) requirements documents. For this purpose, we conduct a preliminary exploratory study considering OpenAI chatGPT-3.5 and Microsoft Bing. Using two exemplar NL requirements documents, we compare the variability detection capability of the chatbots with that of experts and that of a rule-based NLP tool.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
We cannot compare with the referenced rule-based tool because it only detects variation points and it is not able to identify features.
- 2.
- 3.
In the form we have also asked students for an analysis exercise of the results, but it was just a classroom assignment, not used for the purpose of this paper.
References
Bach, S.H., et al. PromptSource: an integrated development environment and repository for natural language prompts. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, ACL 2022 - System Demonstrations, Dublin, Ireland, pp. 93–104. Association for Computational Linguistics, 22–27 May (2022)
Bakar, N.H., Kasirun, Z.M., Salleh, N.: Feature extraction approaches from natural language requirements for reuse in software product lines: a systematic literature review. J. Syst. Softw. 106, 132–149 (2015)
Brown, T.B., Mann, B., et al.: Language models are few-shot learners. In: 33rd Annual Conference on Neural Information Processing Systems, 6–12 Dec (2020)
Clements, P., Northrop, L.M.: Software product lines-practices and patterns. SEI series in Software Engineering. Addison-Wesley (2002)
Fantechi, A., Gnesi, S., Semini, L.: VIBE: looking for variability in ambiguous requirements. J. Syst. Softw. 195 (2023)
Galindo, J.A., Dominguez, A.J., White, J., Benavides, D.: Large language models to generate meaningful feature model instances. In: Proceedings of the 27th ACM International Systems and Software Product Line Conference - Volume A, SPLC 2023, pp. 15–26. Association for Computing Machinery, New York (2023)
Itzik, N., Reinhartz-Berger, I., Wand, Y.: Variability analysis of requirements: considering behavioral differences and reflecting stakeholders’ perspectives. TSE 42, 687–706 (2016)
Li, Y., Schulze, S., Saake, G.: Reverse engineering variability from natural language documents: a systematic literature review. In: Proceedings of the 21st International Systems and Software Product Line Conference, SPLC, pp. 133–142. ACM (2017)
Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., Neubig, G.: Pre-train, prompt, and predict: a systematic survey of prompting methods in natural language processing. ACM Comput. Surv. 55(9), 195:1–195:35 (2023)
Mefteh, H.B.-A.M., Bouassida, N.: Mining feature models from functional requirements. Comput. J. 59, 1784–1804 (2016)
Pohl, K., Böckle, G., van der Linden, F.: Software Product Line Engineering - Foundations, Principles, and Techniques. Springer (2005). https://doi.org/10.1007/3-540-28901-1
Schulze, S., Li, Y.: Feature and variability extraction from natural language requirements. In: Lopez-Herrejon, R.E., Martinez, J., Assunção, W.K.G., Ziadi, T., Acher, M., Vergilio, S.R. (eds.) Handbook of Re-Engineering Software Intensive Systems into Software Product Lines, pp. 31–52. Springer International Publishing (2023). https://doi.org/10.1007/978-3-031-11686-5_2
Sree-Kumar, A., Planas, E., Clarisó, R.: Extracting software product line feature models from natural language specifications. In: Proceeedings of the 22nd International Systems and Software Product Line Conference - Volume 1, SPLC 2018, Gothenburg, Sweden, 10–14 September, pp. 43–53. ACM (2018)
White, J., et al.: A prompt pattern catalog to enhance prompt engineering with chatgpt. CoRR, abs/ arXiv: 2302.11382 (2023)
Acknowledgements
The research has been partially supported by the MIUR, Italy project PRIN 2022 STENDHAL. We gratefully thank the reviewers for their thoughtful comments and the students of the software engineering course in Pisa for their help.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Fantechi, A., Gnesi, S., Semini, L. (2024). Exploring LLMs’ Ability to Detect Variability in Requirements. In: Mendez, D., Moreira, A. (eds) Requirements Engineering: Foundation for Software Quality. REFSQ 2024. Lecture Notes in Computer Science, vol 14588. Springer, Cham. https://doi.org/10.1007/978-3-031-57327-9_11
Download citation
DOI: https://doi.org/10.1007/978-3-031-57327-9_11
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-57326-2
Online ISBN: 978-3-031-57327-9
eBook Packages: Computer ScienceComputer Science (R0)