Skip to main content
SpringerLink
Log in

Exploring LLMs’ Ability to Detect Variability in Requirements

  • Conference paper
  • First Online:
Requirements Engineering: Foundation for Software Quality (REFSQ 2024)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14588))

  • 191 Accesses

Abstract

In this paper, we address the question of whether general-purpose LLM-based tools may be useful for detecting requirements variability in Natural Language (NL) requirements documents. For this purpose, we conduct a preliminary exploratory study considering OpenAI chatGPT-3.5 and Microsoft Bing. Using two exemplar NL requirements documents, we compare the variability detection capability of the chatbots with that of experts and that of a rule-based NLP tool.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Institutional subscriptions

Notes

  1. 1.

    We cannot compare with the referenced rule-based tool because it only detects variation points and it is not able to identify features.

  2. 2.

    https://docs.google.com/forms/d/e/1FAIpQLSfscV_uiATaWSJngBH9ruXJBMAnJDvtw6TVLOMXgToFZQ1n8Q/viewform.

  3. 3.

    In the form we have also asked students for an analysis exercise of the results, but it was just a classroom assignment, not used for the purpose of this paper.

References

  1. Bach, S.H., et al. PromptSource: an integrated development environment and repository for natural language prompts. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, ACL 2022 - System Demonstrations, Dublin, Ireland, pp. 93–104. Association for Computational Linguistics, 22–27 May (2022)

    Google Scholar 

  2. Bakar, N.H., Kasirun, Z.M., Salleh, N.: Feature extraction approaches from natural language requirements for reuse in software product lines: a systematic literature review. J. Syst. Softw. 106, 132–149 (2015)

    Article  Google Scholar 

  3. Brown, T.B., Mann, B., et al.: Language models are few-shot learners. In: 33rd Annual Conference on Neural Information Processing Systems, 6–12 Dec (2020)

    Google Scholar 

  4. Clements, P., Northrop, L.M.: Software product lines-practices and patterns. SEI series in Software Engineering. Addison-Wesley (2002)

    Google Scholar 

  5. Fantechi, A., Gnesi, S., Semini, L.: VIBE: looking for variability in ambiguous requirements. J. Syst. Softw. 195 (2023)

    Google Scholar 

  6. Galindo, J.A., Dominguez, A.J., White, J., Benavides, D.: Large language models to generate meaningful feature model instances. In: Proceedings of the 27th ACM International Systems and Software Product Line Conference - Volume A, SPLC 2023, pp. 15–26. Association for Computing Machinery, New York (2023)

    Google Scholar 

  7. Itzik, N., Reinhartz-Berger, I., Wand, Y.: Variability analysis of requirements: considering behavioral differences and reflecting stakeholders’ perspectives. TSE 42, 687–706 (2016)

    Google Scholar 

  8. Li, Y., Schulze, S., Saake, G.: Reverse engineering variability from natural language documents: a systematic literature review. In: Proceedings of the 21st International Systems and Software Product Line Conference, SPLC, pp. 133–142. ACM (2017)

    Google Scholar 

  9. Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., Neubig, G.: Pre-train, prompt, and predict: a systematic survey of prompting methods in natural language processing. ACM Comput. Surv. 55(9), 195:1–195:35 (2023)

    Google Scholar 

  10. Mefteh, H.B.-A.M., Bouassida, N.: Mining feature models from functional requirements. Comput. J. 59, 1784–1804 (2016)

    Article  Google Scholar 

  11. Pohl, K., Böckle, G., van der Linden, F.: Software Product Line Engineering - Foundations, Principles, and Techniques. Springer (2005). https://doi.org/10.1007/3-540-28901-1

  12. Schulze, S., Li, Y.: Feature and variability extraction from natural language requirements. In: Lopez-Herrejon, R.E., Martinez, J., Assunção, W.K.G., Ziadi, T., Acher, M., Vergilio, S.R. (eds.) Handbook of Re-Engineering Software Intensive Systems into Software Product Lines, pp. 31–52. Springer International Publishing (2023). https://doi.org/10.1007/978-3-031-11686-5_2

  13. Sree-Kumar, A., Planas, E., Clarisó, R.: Extracting software product line feature models from natural language specifications. In: Proceeedings of the 22nd International Systems and Software Product Line Conference - Volume 1, SPLC 2018, Gothenburg, Sweden, 10–14 September, pp. 43–53. ACM (2018)

    Google Scholar 

  14. White, J., et al.: A prompt pattern catalog to enhance prompt engineering with chatgpt. CoRR, abs/ arXiv: 2302.11382 (2023)

Download references

Acknowledgements

The research has been partially supported by the MIUR, Italy project PRIN 2022 STENDHAL. We gratefully thank the reviewers for their thoughtful comments and the students of the software engineering course in Pisa for their help.

Author information

Authors and Affiliations

  1. Dipartimento di Informatica, Università di Pisa, Pisa, Italy

    Laura Semini

  2. Dipartimento di Ingegneria dell’Informazione, Università di Firenze, Florence, Italy

    Alessandro Fantechi

  3. Istituto di Scienza e Tecnologie dell’Informazione “A. Faedo”, ISTI-CNR, Pisa, Italy

    Alessandro Fantechi, Stefania Gnesi & Laura Semini

Authors
  1. Alessandro Fantechi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Stefania Gnesi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Laura Semini
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Laura Semini .

Editor information

Editors and Affiliations

  1. Blekinge Institute of Technology, Karlskrona, Sweden

    Daniel Mendez

  2. NOVA University Lisbon, Lisbon, Portugal

    Ana Moreira

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Fantechi, A., Gnesi, S., Semini, L. (2024). Exploring LLMs’ Ability to Detect Variability in Requirements. In: Mendez, D., Moreira, A. (eds) Requirements Engineering: Foundation for Software Quality. REFSQ 2024. Lecture Notes in Computer Science, vol 14588. Springer, Cham. https://doi.org/10.1007/978-3-031-57327-9_11

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-57327-9_11

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-57326-2

  • Online ISBN: 978-3-031-57327-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation