Skip to main content
SpringerLink
Log in

Generating domain models from natural language text using NLP: a benchmark dataset and experimental comparison of tools

  • Theme Section Paper
  • Published:
Software and Systems Modeling Aims and scope Submit manuscript

Abstract

Software requirements specification describes users’ needs and expectations on some target system. Requirements documents are typically represented by unstructured natural language text. Such texts are the basis for the various subsequent activities in software development, such as software analysis and design. As part of software analysis, domain models are made that describe the key concepts and relations between them. Since the analysis process is performed manually by business analysts, it is time-consuming and may introduce mistakes. Recently, researchers have worked toward automating the synthesis of domain models from textual software requirements. Current studies on this topic have limitations in terms of the volume and heterogeneity of experimental datasets. To remedy this, we provide a curated dataset of software requirements to be utilized as a benchmark by algorithms that transform textual requirements documents into domain models. We present a detailed evaluation of two text-to-model approaches: one based on a large-language model (ChatGPT) and one building on grammatical rules (txt2Model). Our evaluation reveals that both tools yield promising results with relatively high F-scores for modeling the classes, attributes, methods, and relationships, with txt2Model performing better than ChatGPT on average. Both tools have relatively lower performance and high variance when it comes to the relation types. We believe our dataset and experimental evaluation pave to way to advance the field of automated model generation from requirements.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

Notes

  1. https://ieee-dataport.org/documents/dataset-text-requirements-models.

  2. https://ieee-dataport.org/documents/dataset-text-requirements-models.

References

  1. Rumbaugh, J., Blaha, M., Premerlani, W., Eddy, F., Lorensen, W.E., et al.: Objectoriented Modeling and Design, vol. 199. Prentice-hall Englewood Cliffs, NJ (1991)

  2. Sagar, V.B.R.V., Abirami, S.: Conceptual modeling of natural language functional requirements. J. Syst. Softw. 88, 25–41 (2014)

    Article  Google Scholar 

  3. Özdağoğlu, A., Özdağoğlu, G.: Comparison of ahp and fuzzy ahp for the multi-criteria decision making processes with linguistic evaluations. İstanbul Ticaret Üniversitesi Fen Bilimleri Dergisi 6(11), 65–85 (2007)

    Google Scholar 

  4. Landhäußer, M., Körner, S.J., Tichy, W.F.: From requirements to UML models and back: How automatic processing of text can support requirements engineering. Softw. Qual. J. 22, 121–149 (2014)

    Article  Google Scholar 

  5. Bozyigit, F., Aktaş, Ö., Kılınç, D.: Linking software requirements and conceptual models: a systematic literature review. Int. J. Eng. Sci. Technol. 24(1), 71–82 (2021)

    Google Scholar 

  6. Satapathy, S.C., Jena, A.K., Singh, J., Bilgaiyan, S.: Automated Software Engineering: A Deep Learning-Based Approach. Springer (2020)

  7. Jadhav, A., Kaur, M., Akter, F.: Evolution of software development effort and cost estimation techniques: five decades study using automated text mining approach. Math. Probl. Eng. 2022, 1–17 (2022)

    Article  Google Scholar 

  8. Mahmood, Y., Kama, N., Azmi, A., Khan, A.S., Ali, M.: Software effort estimation accuracy prediction of machine learning techniques: a systematic performance evaluation. Software Practice and Experience 52(1), 39–65 (2022)

    Article  Google Scholar 

  9. Giray, G., Bennin, K.E., Köksal, Ö., Babur, Ö., Tekinerdogan, B.: On the use of deep learning in software defect prediction. J. Syst. Softw. 195, 111–537 (2023)

    Article  Google Scholar 

  10. Khalilipour, A., Bozyigit, F., Utku, C., Challenger, M.: Machine learning based model categorization using textual and structural features. In: European Conference on Advances in Databases and Information Systems. Springer, pp. 425–436 (2022)

  11. Khalilipour, A., Bozyigit, F., Utku, C., Challenger, M.: Categorization of the models based on structural information extraction and machine learning. In; International Conference on Intelligent and Fuzzy Systems. Springer, pp. 173–181 (2022)

  12. Rahimi, S., Lano, K.C., Lin, C.: Requirement formalisation using natural language processing and machine learning: A systematic review. In: International conference on Model-Based Software and Systems Engineering, SCITEPRESS Digital Library, pp. 1–8 (2022)

  13. Zhao, L., et al.: Natural language processing for requirements engineering: a systematic mapping study. ACM Comput. Surv. 54(3), 1–41 (2021)

    Article  Google Scholar 

  14. Falessi, D., Cantone, G.: The effort savings from using NLP to classify equivalent requirements. IEEE Softw. 36(1), 48–55 (2018)

    Article  Google Scholar 

  15. Li, G., Zheng, C., Li, M., Wang, H.: Automatic requirements classification based on graph attention network. IEEE Access 10, 30080–30090 (2022)

    Article  Google Scholar 

  16. Ahmed, S., Ahmed, A., Eisty, N.U.: Automatic transformation of natural to unified modeling language: a systematic review. In: 2022 IEEE/ACIS 20th International Conference on Software Engineering Research, Management and Applications (SERA), IEEE, pp. 112–119 (2022)

  17. Habibullah, K.M., Gay, G., Horkoff, J.: Non-functional requirements for machine learning: Understanding current use and challenges among practitioners. Requirem. Eng. pp. 1–34 (2023)

  18. Liu, Z., Lin, Y., Sun, M., Liu, Z., Lin, Y.: Representation learning and NLP. Representation Learning for Natural Language Processing, pp. 1–11 (2020)

  19. Mich, L.: Nl-oops: From natural language to object oriented requirements using the natural language processing system lolita. Nat. Lang. Eng. 2(2), 161–187 (1996)

    Article  Google Scholar 

  20. Ibrahim, M., Ahmad, R.: Class diagram extraction from textual requirements using natural language processing (nlp) techniques. In: Second International Conference on Computer Research and Development, pp. 200–204 (2010). https://doi.org/10.1109/ICCRD.2010.71

  21. Zhou, X., Zhou, N., Zhou, N.: Auto-generation of class diagram from free-text functional specifications and domain ontology (2004)

  22. Bajwa, I.S.: Object oriented software modeling using NLP based knowledge extraction (2009)

  23. Tripathy, A., Agrawal, A., Rath, S.K.: Requirement analysis using natural language processing. In: Fifth International Conference on Advances in Computer Engineering, vol. 26, p. 27 (2014)

  24. Dori, D., Korda, N., Soffer, A., Cohen, S.: Smart: system model acquisition from requirements text. In: Proceedings of Business Process Management: Second International Conference, BPM: Potsdam, Germany, pp. 179–194. Springer (2004)

  25. Abdelnabi, E.A., Maatuk, A.M., Abdelaziz, T.M., Elakeili, S.M.: Generating UML class diagram using nlp techniques and heuristic rules. In: 2020 20th International Conference on Sciences and Techniques of Automatic Control and Computer Engineering (STA), IEEE, pp. 277–282 (2020)

  26. Deeptimahanti, D.K., Babar, M.A.: An automated tool for generating UML models from natural language requirements. In: 2009 IEEE/ACM International Conference on Automated Software Engineering, IEEE, pp. 680–682 (2009)

  27. Robeer, M., Lucassen, G., Van DerWerf, J.M.E., Dalpiaz, F., Brinkkemper, S.: Automated extraction of conceptual models from user stories via NLP. In: IEEE 24th International Requirements Engineering Conference (RE), pp. 196–205. IEEE (2016)

  28. Lucassen, G., Robeer, M., Dalpiaz, F., Van Der Werf, J.M.E., Brinkkemper, S.: Extracting conceptual models from user stories with visual narrator. Requir. Eng. 22, 339–358 (2017)

    Article  Google Scholar 

  29. Bragilovski, M., Dalpiaz, F., Sturm, A.: From user stories to domain models: recommending relationships between entities (2023)

  30. Ferrari, A., Spagnolo, G.O., Gnesi, S.: Pure: a dataset of public requirements documents. In: IEEE 25th International Requirements Engineering Conference (RE), pp. 502–505. IEEE (2017)

  31. Ramackers, G.J., Griffioen, P.P., Schouten, M.B., Chaudron, M.R.V.: From prose to prototype: synthesising executable UML models from natural language. In: 2021 ACM/IEEE International Conference on Model Driven Engineering Languages and Systems Companion (MODELS-C), pp. 380–389. IEEE (2021)

  32. Sedrakyan, G., Abdi, A., Van Den Berg, S.M., Veldkamp, B.P., Van Hillegersberg, J.: Text-to-model (tetomo) transformation framework to support requirements analysis and modeling. In: MODELSWARD, pp. 129–136 (2022)

  33. Hamza, Z.A., Hammad, M.: Generating UML use case models from software requirements using natural language processing. In: 2019 8th International Conference on Modeling Simulation and Applied Optimization (ICMSAO), pp. 1–6, IEEE (2019)

  34. Arora, c., Sabetzadeh, M., Briand, L., Zimmer, F.: Extracting domain models from natural-language requirements: approach and industrial evaluation. In: Proceedings of the ACM/IEEE 19th International Conference on Model Driven Engineering Languages and Systems, pp. 250–260 (2016)

  35. Cámara, J., Troya, J., Burgueño, L., Vallecillo, A.: On the assessment of generative AI in modeling tasks: an experience report with chatgpt and uml. Softw. Syst. Model, pp. 1–13 (2023)

  36. Lano, K., Yassipour-Tehrani, S., Umar, M.: Automated requirements formalisation for agile MDE. In 2021 ACM/IEEE International Conference on Model Driven Engineering Languages and Systems Companion (MODELS-C), pp. 173–180. IEEE (2021)

  37. Bozyigit, F., Aktaş, Ö., Kılınç, D.: Automatic concept identification of software requirements in Turkish. Turkish Journal of Electrical Engineering and Computer Sciences (2019)

  38. Wu, T., et al.: A brief overview of chatgpt: the history, status quo and potential future development. IEEE/CAA Journal of Automatica Sinica 10(5), 1122–1136 (2023)

    Article  Google Scholar 

  39. Sim, S.E., Easterbrook, S., Holt, R.C.: Using benchmarking to advance research: a challenge to software engineering. In: Proceedings of 25th International Conference on Software Engineering, pp. 74–83. IEEE (2003)

  40. Kistowski, J.v., Arnold, J.A., Huppler, K., Lange, K.-D., Henning, J.L., Cao, P.: How to build a benchmark. In: Proceedings of the 6th ACM/SPEC International Conference on Performance Engineering, pp. 333–336 (2015)

  41. Bozyigit, F., Bardakci, T., Khalilipour, A., Challenger, M., Ramackers, G., Babur, O., Chaudron, M.R.V.: Dataset for: Text requirements to models. (2023). https://doi.org/10.21227/r9j6-nd62

  42. Brereton, P., Kitchenham, B.A., Budgen, D., Turner, M., Khalil, M.: Lessons from applying the systematic literature review process within the software engineering domain. J. Syst. Softw. 80(4), 571–583 (2007). https://doi.org/10.1016/j.jss.2006.07.009

    Article  Google Scholar 

Download references

Acknowledgements

We acknowledge the sharing of cases by Jörg Kienzle and Günter Mussbacher. Ultimately the cases they provided were not included in our final dataset.

Author information

Author notes

  1. Tolgahan Bardakci, Alireza Khalilipour, Moharram Challenger, Guus Ramackers, Önder Babur, and Michel R. V. Chaudron have contributed equally to this work.

Authors and Affiliations

  1. Department of Computer Science, University of Antwerp, Antwerp, Belgium

    Fatma Bozyigit, Tolgahan Bardakci, Alireza Khalilipour & Moharram Challenger

  2. AnSyMo/Cosys core lab, Flanders Make Strategic Research Center, Lommel, Belgium

    Fatma Bozyigit, Alireza Khalilipour & Moharram Challenger

  3. Leiden Institute of Advanced Computer Science (LIACS), Leiden University, Leiden, The Netherlands

    Guus Ramackers

  4. Information Technology Group, Wageningen University and Research, Wageningen, The Netherlands

    Önder Babur

  5. Department of Mathematics and Computer Science, Eindhoven University of Technology, Eindhoven, The Netherlands

    Önder Babur & Michel R. V. Chaudron

Authors
  1. Fatma Bozyigit
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Tolgahan Bardakci
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Alireza Khalilipour
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Moharram Challenger
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Guus Ramackers
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Önder Babur
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Michel R. V. Chaudron
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Fatma Bozyigit.

Additional information

Communicated by Lano, Kolahdouz-Rahimi, Yassipour-Tehrani, Burgueño, and Uma.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bozyigit, F., Bardakci, T., Khalilipour, A. et al. Generating domain models from natural language text using NLP: a benchmark dataset and experimental comparison of tools. Softw Syst Model (2024). https://doi.org/10.1007/s10270-024-01176-y

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10270-024-01176-y

Keywords

Search

Navigation