Skip to main content
SpringerLink
Log in

Do software models based on the UML aid in source-code comprehensibility? Aggregating evidence from 12 controlled experiments

  • Published:
Empirical Software Engineering Aims and scope Submit manuscript

Abstract

In this paper, we present the results of long-term research conducted in order to study the contribution made by software models based on the Unified Modeling Language (UML) to the comprehensibility of Java source-code deprived of comments. We have conducted 12 controlled experiments in different experimental contexts and on different sites with participants with different levels of expertise (i.e., Bachelor’s, Master’s, and PhD students and software practitioners from Italy and Spain). A total of 333 observations were obtained from these experiments. The UML models in our experiments were those produced in the analysis and design phases. The models produced in the analysis phase were created with the objective of abstracting the environment in which the software will work (i.e., the problem domain), while those produced in the design phase were created with the goal of abstracting implementation aspects of the software (i.e., the solution/application domain). Source-code comprehensibility was assessed with regard to correctness of understanding, time taken to accomplish the comprehension tasks, and efficiency as regards accomplishing those tasks. In order to study the global effect of UML models on source-code comprehensibility, we aggregated results from the individual experiments using a meta-analysis. We made every effort to account for the heterogeneity of our experiments when aggregating the results obtained from them. The overall results suggest that the use of UML models affects the comprehensibility of source-code, when it is deprived of comments. Indeed, models produced in the analysis phase might reduce source-code comprehensibility, while increasing the time taken to complete comprehension tasks. That is, browsing source code and this kind of models together negatively impacts on the time taken to complete comprehension tasks without having a positive effect on the comprehensibility of source code. One plausible justification for this is that the UML models produced in the analysis phase focus on the problem domain. That is, models produced in the analysis phase say nothing about source code and there should be no expectation that they would, in any way, be beneficial to comprehensibility. On the other hand, UML models produced in the design phase improve source-code comprehensibility. One possible justification for this result is that models produced in the design phase are more focused on implementation details. Therefore, although the participants had more material to read and browse, this additional effort was paid back in the form of an improved comprehension of source code.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

Notes

  1. An instance of a process is a sequence of the activities/phases.

  2. It is also called requirements engineering process and it is the process of determining user expectations (i.e., requirements) as regards a new or modified product.

  3. It maps the requirements onto the software architecture that defines the components, their interfaces and behaviors. The design document describes a plan with which to implement the requirements.

  4. Design-pattern instances can be seen as a micro-architecture that developers copy and adapt to their particular designs in order to solve the recurrent problem described by the design pattern (Bruegge and Dutoit 2003; Gamma et al. 1995).

  5. www2.unibas.it/gscanniello/SourceCodeComprMetaAnalysis/data.xlsx

  6. This is a software system that is used to sell and manage CDs/DVDs in a music shop. The feature search for a singer was used in the experiments: the user inserts a string (e.g., the surname of the singer), and the system then searches for all the singers that satisfy the search criterion and shows them in a list of the associated information.

  7. This is a software system with which to book and buy theater tickets. The feature buy a theater ticket was used in the experiments: the system shows the list of the available tickets for a given theater and performance, and the user then chooses the ticket and inserts data about the spectator.

  8. The time was an approximation of comprehension effort. This complies with the ISO/IEC 9126 standard ISO (1991) (and subsequent versions), in which effort is the productive time associated with a specific project task.

  9. If a participant is tested first under the experimental condition A and then under the experimental condition B, she/he could potentially perform better or worse under condition B.

  10. Efficiency is a derived measure that is computed as the ratio between comprehension and completion time. Task efficiency is a ratio measure and estimates a participant’s efficiency as regards the execution of a comprehension task. The larger the efficiency value, the better it is. The perspective we adopted is that of quality in use (e.g., ISO 2000, 2011), since efficiency measures source-code comprehension achieved during the expenditure of available models.

  11. Effect size is statistically different from the overall effect if the diamond (at the bottom of the forest plot) does not intersect the vertical line.

  12. This is a range of values that we are 95% certain that it contains the true mean value.

References

  • Abrahão SM, Gravino C, Pelozo EI, Scanniello G, Tortora G (2013) Assessing the effectiveness of sequence diagrams in the comprehension of functional requirements: Results from a family of five experiments. IEEE Trans Softw Eng 39 (3):327–342

    Article  Google Scholar 

  • Agarwal R, Sinha AP (2003) Object-oriented modeling with UML: a study of developers’ perceptions. Commun ACM 46(9):248–256

    Article  Google Scholar 

  • Anda B, Hansen K, Gullesen I, Thorsen HK (2006) Experiences from introducing UML-based development in a large safety-critical project. Empir Softw Eng 11(4):555–581

    Article  Google Scholar 

  • Antoniol G, Canfora G, Casazza G, De Lucia A, Merlo E (2002) Recovering traceability links between code and documentation. IEEE Trans Softw Eng 28(10):970–983

    Article  Google Scholar 

  • Arisholm E, Briand LC, Hove SE, Labiche Y (2006) The impact of UML documentation on software maintenance: An experimental evaluation. IEEE Trans Softw Eng 32(6):365–381

    Article  Google Scholar 

  • Basili VR, Rombach HD (1988) The TAME project: Towards improvement-oriented software environments. IEEE Trans Softw Eng 14(6):758–773

    Article  Google Scholar 

  • Basili V, Shull F, Lanubile F (1999) Building knowledge through families of experiments. IEEE Trans Softw Eng 25(4):456–473

    Article  Google Scholar 

  • Bavota G, Canfora G, Di Penta M, Oliveto R, Panichella S (2013) An empirical investigation on documentation usage patterns in maintenance tasks. In: Proceedings of International Conference on Software Maintenance. IEEE Computer Society, pp 210–219

  • Bruegge B, Dutoit AH (2003) Object-oriented software engineering: using UML, Patterns and Java, 2nd edn. Prentice-Hall, Upper Saddle River

    Google Scholar 

  • Budgen D, Burn AJ, Brereton OP, Kitchenham B, Pretorius R (2011) Empirical evidence about the UML: a systematic literature review. Softw Pract Exper 41(4):363–392

    Article  Google Scholar 

  • Cariou E, Beugnard A, Jezequel JM (2002) An architecture and a process for implementing distributed collaborations. In: Proceedings of International Enterprise Distributed Object Computing, pp 132–143

  • Carver J, Jaccheri L, Morasca S, Shull F (2003) Issues in using students in empirical studies in software engineering education. In: Proceedings of International Symposium on Software Metrics. IEEE Computer Society, pp 239–250

  • Corazza A, Maggio V, Scanniello G (2016) Coherence of comments and method implementations: a dataset and an empirical investigation. Softw Q J:1–27. https://doi.org/10.1007/s11219-016-9347-1

  • Dzidek WJ, Arisholm E, Briand LC (2008) A realistic empirical evaluation of the costs and benefits of UML in software maintenance. IEEE Trans Softw Eng 34 (3):407–432

    Article  Google Scholar 

  • Eclipse Modeling Framework (EMF) (2012) http://www.eclipse.org/modeling/emf/

  • Erickson J, Siau K (2007) Theoretical and practical complexity of modeling methods. Commun ACM 50(8):46–51

    Article  Google Scholar 

  • Fernȧndez-Sȧez AM, Chaudron MRV, Genero M (2013) Exploring costs and benefits of using UML on maintenance: Preliminary findings of a case study in a large IT department. In: Proceedings of the International Workshop on Experiences and Empirical Studies in Software Modeling co-located with the International Conference on Model Driven Engineering Languages and Systems, pp 33–42

  • Fernȧndez-Sȧez AM, Caivano D, Genero M, Chaudron MRV (2015) On the use of UML documentation in software maintenance: Results from a survey in industry. In: Proceedings of ACM/IEEE International Conference on Model Driven Engineering Languages and Systems, pp 292–301

  • Fernȧndez-Sȧez AM, Genero M, Caivano D, Chaudron MRV (2016) Does the level of detail of UML diagrams affect the maintainability of source code?: a family of experiments. Empir Softw Eng 21(1):212–259

    Article  Google Scholar 

  • Fernȧndez-Sȧez AM, Genero M, Chaudron MRV (2013) Empirical studies concerning the maintenance of UML diagrams and their use in the maintenance of code: A systematic mapping study. Inf Softw Technol 55(7):1119–1142

    Article  Google Scholar 

  • Fernȧndez-Sȧez AM, Genero M, Chaudron MRV, Caivano D, Ramos I (2015) Are forward designed or reverse-engineered UML diagrams more helpful for code maintenance?: A family of experiments. Inf Softw Technol 57:644–663

    Article  Google Scholar 

  • Fluri B, Wursch M, Gall H (2007) Do code and comments co-evolve? on the relation between source code and comment changes. In: Proceedings of the Working Conference on Reverse Engineering. IEEE Computer Society, pp 70–79

  • Fu R, Gartlehner G, Grant M, Shamliyan T, Sedrakyan A, Wilt TJ, Griffith L, Oremus M, Raina P, Ismaila A, Santaguida P, Lau J, Trikalinos TA (2011) Conducting quantitative synthesis when comparing medical interventions: AHRQ and the effective health care program. J Clin Epidemiol 64(11):1187–1197

    Article  Google Scholar 

  • Gamma E, Helm R, Johnson R, Vlissides J (1995) Design patterns: elements of reusable object oriented software. Addison-Wesley, Boston

    MATH  Google Scholar 

  • Garousi G, Garousi V, Moussavi M, Ruhe G, Smith B (2013) Evaluating usage and quality of technical software documentation: an empirical study. In: Proceedings of International Conference on Evaluation and Assessment in Software Engineering, pp 24–35

  • Gravino C, Tortora G, Scanniello G (2010) An empirical investigation on the relation between analysis models and source code comprehension. In: Proceedings of the International Symposium on Applied Computing. ACM, pp 2365–2366

  • Gravino C, Scanniello G, Tortora G (2015) Source-code comprehension tasks supported by UML design models: Results from a controlled experiment and a differentiated replication. J Vis Lang Comput 28:23–38

    Article  Google Scholar 

  • Grossman M, Aronson JE, McCarthy RV (2005) Does UML make the grade? Insights from the software development community. Inf Softw Technol 47(6):383–397

    Article  Google Scholar 

  • Guéhéneuc YG (2007) P-mart: Pattern-like micro architecture repository. In: Proceedings of EuroPLoP Focus Group on Pattern Repositories

  • Hammad M, Collard ML, Maletic JI (2011) Automatically identifying changes that impact code-to-design traceability during evolution. Softw Qual J 19(1):35–64

    Article  Google Scholar 

  • Higgins JPT, Green S (2008) Cochrane handbook for systematic reviews of interventions, 5th edn. The Cochrane Collaboration, London

    Book  Google Scholar 

  • Huedo-Medina TB, Sánchez-Meca J, Marín-Martínez F, Botella J (2006) Assessing heterogeneity in meta-analysis: Q statistic or i2 index? Psychol Methods 11 (2):193–206

    Article  Google Scholar 

  • Hutchinson JE, Whittle J, Rouncefield M, Kristoffersen S (2011) Empirical assessment of MDE in industry. In: Proceedings of the International Conference on Software Engineering, pp 471–480

  • ISO (1991) Information Technology–Software Product Evaluation: Quality Characteristics and Guidelines for their Use ISO/IEC IS 9126. ISO, Geneva

  • ISO (2000) ISO 9241-11: Ergonomic requirements for office work with visual display terminals (VDTs) – Part 9: Requirements for non-keyboard input devices. ISO, Geneva

  • ISO (2011) ISO/IEC 25010 Systems and software engineering – Systems and software Quality Requirements and Evaluation (SQuaRE) – System and software quality models. ISO, Geneva

  • Jedlitschka A, Ciolkowski M, Pfahl D, Sjoberg D (2008) Reporting experiments in software engineering. In: Shull F, Singer J (eds) Guide to Advanced Empirical Software Engineering, Springer, pp 201–228

  • Jiang ZM, Hassan AE (2006) Examining the evolution of code comments in postgresql. In: Proceedings of Mining Software Repositories. ACM, pp 179–180

  • Juristo N, Moreno A (2001) Basics of software engineering experimentation. Kluwer Academic Publishers, Dordrecht

    Book  MATH  Google Scholar 

  • Kitchenham B, Pfleeger S, Pickard L, Jones P, Hoaglin D, El Emam K, Rosenberg J (2002) Preliminary guidelines for empirical research in software engineering. IEEE Trans Softw Eng 28(8):721–734

    Article  Google Scholar 

  • Kitchenham B, Charters S (2007) Guidelines for performing systematic literature reviews in software engineering

  • Lehnert S, Farooq Q u a, Riebisch M (2013) Rule-based impact analysis for heterogeneous software artifacts. In: Proceedings of the European Conference on Software Maintenance and Reengineering, pp 209–218

  • Leotta M, Ricca F, Antoniol G, Garousi V, Zhi J, Ruhe G (2013) A pilot experiment to quantify the effect of documentation accuracy on maintenance tasks. In: Proceedings of International Conference on Software Maintenance, pp 428–431

  • OMG (2005) Unified modeling language (UML) specification version 2.0. Technical report, Object Management Group

  • Oppenheim AN (1992) Questionnaire design, interviewing and attitude measurement, Pinter, London

  • Pavalkis S, Nemuraite L (2013) Process for applying derived property based traceability framework in software and systems development life cycle. Springer, Berlin Heidelberg, pp 122–133

    Google Scholar 

  • Pavalkis S, Nemuraite L, Butkiene R (2013) Derived properties: A user friendly approach to improving model traceability. Inf Technol Control 42(1):48–60

    Google Scholar 

  • Pickard L, Kitchenham B, Jones P (1998) Combining empirical results in software engineering. Inf Softw Technol 40(14):811–821

    Article  Google Scholar 

  • Ried K (2008) Interpreting and understanding meta-analysis graphs - A practical guide, vol 35. Australian College of General Practitioners, East Melbourne

    Google Scholar 

  • Salviulo F, Scanniello G (2014) Dealing with identifiers and comments in source code comprehension and maintenance: Results from an ethnographically-informed study with students and professionals. In: Proceedings of International Conference on Evaluation and Assessment in Software Engineering. ACM

  • Scanniello G, Gravino C, Risi M, Tortora G (2010) A controlled experiment for assessing the contribution of design pattern documentation on software maintenance. In: Proceedings of the Symposium on Empirical Software Engineering and Measurement. ACM

  • Scanniello G, Gravino C, Tortora G (2010) Investigating the role of UML in the software modeling and maintenance - a preliminary industrial survey. In: Proceedings of International Conference on Enterprise Information Systems, pp 141–148

  • Scanniello G, Gravino C, Genero M, Cruz-Lemus JA, Tortora G (2014) On the impact of UML analysis models on source code comprehensibility and modifiability. ACM Trans Softw Eng Methodol 23(2):13:1–13:26

    Article  Google Scholar 

  • Scanniello G, Gravino C, Risi M, Tortora G, Dodero G (2015) Documenting design-pattern instances: A family of experiments on source-code comprehensibility. ACM Trans Softw Eng Methodol 24(3):14

    Article  Google Scholar 

  • Scanniello G, Gravino C, Tortora G, Genero M, Risi M, Cruz-Lemus JA, Dodero G (2015) Studying the effect of uml-based models on source-code comprehensibility: Results from a long-term investigation. In: Springer (ed.) Proceedings of International Conference on Product-Focused Software Process Improvement, vol 9459. Lecture Notes in Computer Science, pp 311–327

  • Settimi R, Cleland-Huang J, Khadra OB, Mody J, Lukasik W, DePalma C (2004) Supporting software evolution through dynamically retrieving traces to uml artifacts. In: Proceedings of International Workshop on Principles of Software Evolution, pp 49–54

  • Shull F, Carver JC, Vegas S, Juzgado NJ (2008) The role of replications in empirical software engineering. Empir Softw Eng 13(2):211–218

    Article  Google Scholar 

  • Sillito J, Murphy GC, De Volder K (2008) Asking and answering questions during a programming change task. IEEE Trans Softw Eng 34(4):434–451

    Article  Google Scholar 

  • Tang A, Jin Y, Han J (2007) A rationale-based architecture model for design traceability and reasoning. J Syst Softw 80(6):918–934

    Article  Google Scholar 

  • Tang A, Nicholson A, Jin Y, Han J (2007) Using bayesian belief networks for change impact analysis in architecture design. J Syst Softw 80(1):127–148

    Article  Google Scholar 

  • Wohlin C, Runeson P, Höst M., Ohlsson M, Regnell B, Wesslén A (2012) Experimentation in software engineering. Springer, Berlin

    Book  MATH  Google Scholar 

  • Zhi J, Sun B, Garousi G, Shahnewaz SM, Ruhe G (2015) Cost, benefits and quality of software development documentation: A systematic mapping. J Syst Softw 99:175–198

    Article  Google Scholar 

Download references

Acknowledgements

The authors would like to thank the participants in the experiments and all the people who supported the research presented in this paper. This work has been partially supported by the SEQUOIA Project, (TIN2015-63502-C3-1-R) (MINECO/FEDER) funded by Fondo Europeo de Desarrollo Regional and Ministerio de Econom/ y Competitividad

Author information

Authors and Affiliations

  1. University of Basilicata, Potenza, Italy

    Giuseppe Scanniello

  2. University of Salerno, Fisciano, Italy

    Carmine Gravino, Genoveffa Tortora & Michele Risi

  3. University of Castilla-La Mancha, Ciudad Real, Spain

    Marcela Genero & José A. Cruz-Lemus

  4. Free University of Bozen, Bolzano-Bozen, Italy

    Gabriella Dodero

Authors
  1. Giuseppe Scanniello
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Carmine Gravino
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Marcela Genero
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. José A. Cruz-Lemus
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Genoveffa Tortora
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Michele Risi
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Gabriella Dodero
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Carmine Gravino.

Additional information

Communicated by: Richard Paige

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Scanniello, G., Gravino, C., Genero, M. et al. Do software models based on the UML aid in source-code comprehensibility? Aggregating evidence from 12 controlled experiments. Empir Software Eng 23, 2695–2733 (2018). https://doi.org/10.1007/s10664-017-9591-4

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10664-017-9591-4

Keywords

Search

Navigation