Skip to main content

Establishing a Benchmark Dataset for Traceability Link Recovery Between Software Architecture Documentation and Models

  • Conference paper
  • First Online:
Software Architecture. ECSA 2022 Tracks and Workshops (ECSA 2022)


In research, evaluation plays a key role to assess the performance of an approach. When evaluating approaches, there is a wide range of possible types of studies that can be used, each with different properties. Benchmarks have the benefit that they establish clearly defined standards and baselines. However, when creating new benchmarks, researchers face various problems regarding the identification of potential data, its mining, as well as the creation of baselines. As a result, some research domains do not have any benchmarks at all. This is the case for traceability link recovery between software architecture documentation and software architecture models. In this paper, we create and describe an open-source benchmark dataset for this research domain. With this benchmark, we define a baseline with a simple approach based on information retrieval techniques. This way, we provide other researchers a way to evaluate and compare their approaches.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
USD 69.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 89.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions


  1. 1.

  2. 2.

  3. 3.

  4. 4.

  5. 5.

  6. 6.

  7. 7.

  8. 8.


  1. Borg, M., Runeson, P., Ardö, A.: Recovering from a decade: a systematic mapping of information retrieval approaches to software traceability. Empir. Softw. Eng. 19(6), 1565–1616 (2013).

    Article  Google Scholar 

  2. Ding, W., Liang, P., Tang, A., Van Vliet, H., Shahin, M.: How do open source communities document software architecture: an exploratory survey. In: 2014 19th International Conference on Engineering of Complex Computer Systems, pp. 136–145 (2014).

  3. Guo, J., Cheng, J., Cleland-Huang, J.: Semantically enhanced software traceability using deep learning techniques. In: Proceedings of the 39th International Conference on Software Engineering, ICSE 2017, pp. 3–14. IEEE Press (2017).

  4. Hebig, R., Quang, T.H., Chaudron, M.R.V., Robles, G., Fernandez, M.A.: The quest for open source projects that use UML: mining github. In: Proceedings of the ACM/IEEE 19th International Conference on Model Driven Engineering Languages and Systems, MODELS 2016, pp. 173–183. Association for Computing Machinery, New York (2016).

  5. Keim, J., Fuchß, D., Corallo, S.: Architecture Documentation Consistency Benchmark (2022).,

  6. Keim, J., Schulz, S., Fuchß, D., Kocher, C., Speit, J., Koziolek, A.: Trace link recovery for software architecture documentation. In: Biffl, S., Navarro, E., Löwe, W., Sirjani, M., Mirandola, R., Weyns, D. (eds.) ECSA 2021. LNCS, vol. 12857, pp. 101–116. Springer, Cham (2021).

    Chapter  Google Scholar 

  7. v. Kistowski, J., Arnold, J.A., Huppler, K., Lange, K.D., Henning, J.L., Cao, P.: How to build a benchmark. In: Proceedings of the 6th ACM/SPEC International Conference on Performance Engineering, ICPE 2015, pp. 333–336. Association for Computing Machinery, New York (2015).

  8. von Kistowski, J., Eismann, S., Schmitt, N., Bauer, A., Grohmann, J., Kounev, S.: TeaStore: a micro-service reference application for benchmarking, modeling and resource management research. In: Proceedings of the 26th IEEE International Symposium on the Modelling, Analysis, and Simulation of Computer and Telecommunication Systems. MASCOTS 2018, September 2018.

  9. Konersmann, M., et al.: Evaluation methods and replicability of software architecture research objects. In: 2022 IEEE 19th International Conference on Software Architecture (ICSA), pp. 157–168 (2022).

  10. Levenshtein, V.I., et al.: Binary codes capable of correcting deletions, insertions, and reversals. In: Soviet physics doklady, vol. 10, pp. 707–710. Soviet Union (1966)

    Google Scholar 

  11. Molenaar, S., Spijkman, T., Dalpiaz, F., Brinkkemper, S.: Explicit alignment of requirements and architecture in agile development. In: Madhavji, N., Pasquale, L., Ferrari, A., Gnesi, S. (eds.) REFSQ 2020. LNCS, vol. 12045, pp. 169–185. Springer, Cham (2020).

    Chapter  Google Scholar 

  12. Rempel, P., Mäder, P.: Estimating the implementation risk of requirements in agile software development projects with traceability metrics. In: Fricker, S.A., Schneider, K. (eds.) REFSQ 2015. LNCS, vol. 9013, pp. 81–97. Springer, Cham (2015).

    Chapter  Google Scholar 

  13. Reussner, R., et al.: The palladio component model. Technical Report 14, Karlsruher Institut für Technologie (KIT) (2011).

  14. Rodriguez, D.V., Carver, D.L.: Multi-objective information retrieval-based NSGA-II optimization for requirements traceability recovery. In: 2020 IEEE International Conference on Electro Information Technology (EIT), pp. 271–280 (2020)., ISSN: 2154-0373

  15. Schröder, S., Riebisch, M.: An ontology-based approach for documenting and validating architecture rules. In: Proceedings of the 12th European Conference on Software Architecture: Companion Proceedings, ECSA 2018, Association for Computing Machinery, New York (2018).

  16. Schulz, S.: Linking software architecture documentation and models. Master’s thesis, Karlsruher Institut für Technologie (KIT) (2020).

  17. Sim, S.E., Easterbrook, S., Holt, R.C.: Using benchmarking to advance research: a challenge to software engineering. In: Proceedings of the 25th International Conference on Software Engineering, ICSE 2003, pp. 74–83. IEEE Computer Society, USA (2003)

    Google Scholar 

  18. Wang, W., Niu, N., Liu, H., Niu, Z.: Enhancing automated requirements traceability by resolving polysemy. In: 2018 IEEE 26th International Requirements Engineering Conference, pp. 40–51 (2018).

  19. Zhang, Y., Wan, C., Jin, B.: An empirical study on recovering requirement-to-code links. In: 2016 17th IEEE/ACIS International Conference on Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing (SNPD), pp. 121–126 (2016).

Download references


This work was supported by funding from the topic Engineering Secure Systems of the Helmholtz Association (HGF) and by KASTEL Security Research Labs. This publication is based on the research project SofDCar, which is funded by the German Federal Ministry for Economic Affairs and Climate Action.

Author information

Authors and Affiliations


Corresponding author

Correspondence to Dominik Fuchß .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Fuchß, D., Corallo, S., Keim, J., Speit, J., Koziolek, A. (2023). Establishing a Benchmark Dataset for Traceability Link Recovery Between Software Architecture Documentation and Models. In: Batista, T., Bureš, T., Raibulet, C., Muccini, H. (eds) Software Architecture. ECSA 2022 Tracks and Workshops. ECSA 2022. Lecture Notes in Computer Science, vol 13928. Springer, Cham.

Download citation

  • DOI:

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-36888-2

  • Online ISBN: 978-3-031-36889-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics