Abstract
In this paper, we present the results of long-term research conducted in order to study the contribution made by software models based on the Unified Modeling Language (UML) to the comprehensibility of Java source-code deprived of comments. We have conducted 12 controlled experiments in different experimental contexts and on different sites with participants with different levels of expertise (i.e., Bachelor’s, Master’s, and PhD students and software practitioners from Italy and Spain). A total of 333 observations were obtained from these experiments. The UML models in our experiments were those produced in the analysis and design phases. The models produced in the analysis phase were created with the objective of abstracting the environment in which the software will work (i.e., the problem domain), while those produced in the design phase were created with the goal of abstracting implementation aspects of the software (i.e., the solution/application domain). Source-code comprehensibility was assessed with regard to correctness of understanding, time taken to accomplish the comprehension tasks, and efficiency as regards accomplishing those tasks. In order to study the global effect of UML models on source-code comprehensibility, we aggregated results from the individual experiments using a meta-analysis. We made every effort to account for the heterogeneity of our experiments when aggregating the results obtained from them. The overall results suggest that the use of UML models affects the comprehensibility of source-code, when it is deprived of comments. Indeed, models produced in the analysis phase might reduce source-code comprehensibility, while increasing the time taken to complete comprehension tasks. That is, browsing source code and this kind of models together negatively impacts on the time taken to complete comprehension tasks without having a positive effect on the comprehensibility of source code. One plausible justification for this is that the UML models produced in the analysis phase focus on the problem domain. That is, models produced in the analysis phase say nothing about source code and there should be no expectation that they would, in any way, be beneficial to comprehensibility. On the other hand, UML models produced in the design phase improve source-code comprehensibility. One possible justification for this result is that models produced in the design phase are more focused on implementation details. Therefore, although the participants had more material to read and browse, this additional effort was paid back in the form of an improved comprehension of source code.
Similar content being viewed by others
Notes
An instance of a process is a sequence of the activities/phases.
It is also called requirements engineering process and it is the process of determining user expectations (i.e., requirements) as regards a new or modified product.
It maps the requirements onto the software architecture that defines the components, their interfaces and behaviors. The design document describes a plan with which to implement the requirements.
This is a software system that is used to sell and manage CDs/DVDs in a music shop. The feature search for a singer was used in the experiments: the user inserts a string (e.g., the surname of the singer), and the system then searches for all the singers that satisfy the search criterion and shows them in a list of the associated information.
This is a software system with which to book and buy theater tickets. The feature buy a theater ticket was used in the experiments: the system shows the list of the available tickets for a given theater and performance, and the user then chooses the ticket and inserts data about the spectator.
The time was an approximation of comprehension effort. This complies with the ISO/IEC 9126 standard ISO (1991) (and subsequent versions), in which effort is the productive time associated with a specific project task.
If a participant is tested first under the experimental condition A and then under the experimental condition B, she/he could potentially perform better or worse under condition B.
Efficiency is a derived measure that is computed as the ratio between comprehension and completion time. Task efficiency is a ratio measure and estimates a participant’s efficiency as regards the execution of a comprehension task. The larger the efficiency value, the better it is. The perspective we adopted is that of quality in use (e.g., ISO 2000, 2011), since efficiency measures source-code comprehension achieved during the expenditure of available models.
Effect size is statistically different from the overall effect if the diamond (at the bottom of the forest plot) does not intersect the vertical line.
This is a range of values that we are 95% certain that it contains the true mean value.
References
Abrahão SM, Gravino C, Pelozo EI, Scanniello G, Tortora G (2013) Assessing the effectiveness of sequence diagrams in the comprehension of functional requirements: Results from a family of five experiments. IEEE Trans Softw Eng 39 (3):327–342
Agarwal R, Sinha AP (2003) Object-oriented modeling with UML: a study of developers’ perceptions. Commun ACM 46(9):248–256
Anda B, Hansen K, Gullesen I, Thorsen HK (2006) Experiences from introducing UML-based development in a large safety-critical project. Empir Softw Eng 11(4):555–581
Antoniol G, Canfora G, Casazza G, De Lucia A, Merlo E (2002) Recovering traceability links between code and documentation. IEEE Trans Softw Eng 28(10):970–983
Arisholm E, Briand LC, Hove SE, Labiche Y (2006) The impact of UML documentation on software maintenance: An experimental evaluation. IEEE Trans Softw Eng 32(6):365–381
Basili VR, Rombach HD (1988) The TAME project: Towards improvement-oriented software environments. IEEE Trans Softw Eng 14(6):758–773
Basili V, Shull F, Lanubile F (1999) Building knowledge through families of experiments. IEEE Trans Softw Eng 25(4):456–473
Bavota G, Canfora G, Di Penta M, Oliveto R, Panichella S (2013) An empirical investigation on documentation usage patterns in maintenance tasks. In: Proceedings of International Conference on Software Maintenance. IEEE Computer Society, pp 210–219
Bruegge B, Dutoit AH (2003) Object-oriented software engineering: using UML, Patterns and Java, 2nd edn. Prentice-Hall, Upper Saddle River
Budgen D, Burn AJ, Brereton OP, Kitchenham B, Pretorius R (2011) Empirical evidence about the UML: a systematic literature review. Softw Pract Exper 41(4):363–392
Cariou E, Beugnard A, Jezequel JM (2002) An architecture and a process for implementing distributed collaborations. In: Proceedings of International Enterprise Distributed Object Computing, pp 132–143
Carver J, Jaccheri L, Morasca S, Shull F (2003) Issues in using students in empirical studies in software engineering education. In: Proceedings of International Symposium on Software Metrics. IEEE Computer Society, pp 239–250
Corazza A, Maggio V, Scanniello G (2016) Coherence of comments and method implementations: a dataset and an empirical investigation. Softw Q J:1–27. https://doi.org/10.1007/s11219-016-9347-1
Dzidek WJ, Arisholm E, Briand LC (2008) A realistic empirical evaluation of the costs and benefits of UML in software maintenance. IEEE Trans Softw Eng 34 (3):407–432
Eclipse Modeling Framework (EMF) (2012) http://www.eclipse.org/modeling/emf/
Erickson J, Siau K (2007) Theoretical and practical complexity of modeling methods. Commun ACM 50(8):46–51
Fernȧndez-Sȧez AM, Chaudron MRV, Genero M (2013) Exploring costs and benefits of using UML on maintenance: Preliminary findings of a case study in a large IT department. In: Proceedings of the International Workshop on Experiences and Empirical Studies in Software Modeling co-located with the International Conference on Model Driven Engineering Languages and Systems, pp 33–42
Fernȧndez-Sȧez AM, Caivano D, Genero M, Chaudron MRV (2015) On the use of UML documentation in software maintenance: Results from a survey in industry. In: Proceedings of ACM/IEEE International Conference on Model Driven Engineering Languages and Systems, pp 292–301
Fernȧndez-Sȧez AM, Genero M, Caivano D, Chaudron MRV (2016) Does the level of detail of UML diagrams affect the maintainability of source code?: a family of experiments. Empir Softw Eng 21(1):212–259
Fernȧndez-Sȧez AM, Genero M, Chaudron MRV (2013) Empirical studies concerning the maintenance of UML diagrams and their use in the maintenance of code: A systematic mapping study. Inf Softw Technol 55(7):1119–1142
Fernȧndez-Sȧez AM, Genero M, Chaudron MRV, Caivano D, Ramos I (2015) Are forward designed or reverse-engineered UML diagrams more helpful for code maintenance?: A family of experiments. Inf Softw Technol 57:644–663
Fluri B, Wursch M, Gall H (2007) Do code and comments co-evolve? on the relation between source code and comment changes. In: Proceedings of the Working Conference on Reverse Engineering. IEEE Computer Society, pp 70–79
Fu R, Gartlehner G, Grant M, Shamliyan T, Sedrakyan A, Wilt TJ, Griffith L, Oremus M, Raina P, Ismaila A, Santaguida P, Lau J, Trikalinos TA (2011) Conducting quantitative synthesis when comparing medical interventions: AHRQ and the effective health care program. J Clin Epidemiol 64(11):1187–1197
Gamma E, Helm R, Johnson R, Vlissides J (1995) Design patterns: elements of reusable object oriented software. Addison-Wesley, Boston
Garousi G, Garousi V, Moussavi M, Ruhe G, Smith B (2013) Evaluating usage and quality of technical software documentation: an empirical study. In: Proceedings of International Conference on Evaluation and Assessment in Software Engineering, pp 24–35
Gravino C, Tortora G, Scanniello G (2010) An empirical investigation on the relation between analysis models and source code comprehension. In: Proceedings of the International Symposium on Applied Computing. ACM, pp 2365–2366
Gravino C, Scanniello G, Tortora G (2015) Source-code comprehension tasks supported by UML design models: Results from a controlled experiment and a differentiated replication. J Vis Lang Comput 28:23–38
Grossman M, Aronson JE, McCarthy RV (2005) Does UML make the grade? Insights from the software development community. Inf Softw Technol 47(6):383–397
Guéhéneuc YG (2007) P-mart: Pattern-like micro architecture repository. In: Proceedings of EuroPLoP Focus Group on Pattern Repositories
Hammad M, Collard ML, Maletic JI (2011) Automatically identifying changes that impact code-to-design traceability during evolution. Softw Qual J 19(1):35–64
Higgins JPT, Green S (2008) Cochrane handbook for systematic reviews of interventions, 5th edn. The Cochrane Collaboration, London
Huedo-Medina TB, Sánchez-Meca J, Marín-Martínez F, Botella J (2006) Assessing heterogeneity in meta-analysis: Q statistic or i2 index? Psychol Methods 11 (2):193–206
Hutchinson JE, Whittle J, Rouncefield M, Kristoffersen S (2011) Empirical assessment of MDE in industry. In: Proceedings of the International Conference on Software Engineering, pp 471–480
ISO (1991) Information Technology–Software Product Evaluation: Quality Characteristics and Guidelines for their Use ISO/IEC IS 9126. ISO, Geneva
ISO (2000) ISO 9241-11: Ergonomic requirements for office work with visual display terminals (VDTs) – Part 9: Requirements for non-keyboard input devices. ISO, Geneva
ISO (2011) ISO/IEC 25010 Systems and software engineering – Systems and software Quality Requirements and Evaluation (SQuaRE) – System and software quality models. ISO, Geneva
Jedlitschka A, Ciolkowski M, Pfahl D, Sjoberg D (2008) Reporting experiments in software engineering. In: Shull F, Singer J (eds) Guide to Advanced Empirical Software Engineering, Springer, pp 201–228
Jiang ZM, Hassan AE (2006) Examining the evolution of code comments in postgresql. In: Proceedings of Mining Software Repositories. ACM, pp 179–180
Juristo N, Moreno A (2001) Basics of software engineering experimentation. Kluwer Academic Publishers, Dordrecht
Kitchenham B, Pfleeger S, Pickard L, Jones P, Hoaglin D, El Emam K, Rosenberg J (2002) Preliminary guidelines for empirical research in software engineering. IEEE Trans Softw Eng 28(8):721–734
Kitchenham B, Charters S (2007) Guidelines for performing systematic literature reviews in software engineering
Lehnert S, Farooq Q u a, Riebisch M (2013) Rule-based impact analysis for heterogeneous software artifacts. In: Proceedings of the European Conference on Software Maintenance and Reengineering, pp 209–218
Leotta M, Ricca F, Antoniol G, Garousi V, Zhi J, Ruhe G (2013) A pilot experiment to quantify the effect of documentation accuracy on maintenance tasks. In: Proceedings of International Conference on Software Maintenance, pp 428–431
OMG (2005) Unified modeling language (UML) specification version 2.0. Technical report, Object Management Group
Oppenheim AN (1992) Questionnaire design, interviewing and attitude measurement, Pinter, London
Pavalkis S, Nemuraite L (2013) Process for applying derived property based traceability framework in software and systems development life cycle. Springer, Berlin Heidelberg, pp 122–133
Pavalkis S, Nemuraite L, Butkiene R (2013) Derived properties: A user friendly approach to improving model traceability. Inf Technol Control 42(1):48–60
Pickard L, Kitchenham B, Jones P (1998) Combining empirical results in software engineering. Inf Softw Technol 40(14):811–821
Ried K (2008) Interpreting and understanding meta-analysis graphs - A practical guide, vol 35. Australian College of General Practitioners, East Melbourne
Salviulo F, Scanniello G (2014) Dealing with identifiers and comments in source code comprehension and maintenance: Results from an ethnographically-informed study with students and professionals. In: Proceedings of International Conference on Evaluation and Assessment in Software Engineering. ACM
Scanniello G, Gravino C, Risi M, Tortora G (2010) A controlled experiment for assessing the contribution of design pattern documentation on software maintenance. In: Proceedings of the Symposium on Empirical Software Engineering and Measurement. ACM
Scanniello G, Gravino C, Tortora G (2010) Investigating the role of UML in the software modeling and maintenance - a preliminary industrial survey. In: Proceedings of International Conference on Enterprise Information Systems, pp 141–148
Scanniello G, Gravino C, Genero M, Cruz-Lemus JA, Tortora G (2014) On the impact of UML analysis models on source code comprehensibility and modifiability. ACM Trans Softw Eng Methodol 23(2):13:1–13:26
Scanniello G, Gravino C, Risi M, Tortora G, Dodero G (2015) Documenting design-pattern instances: A family of experiments on source-code comprehensibility. ACM Trans Softw Eng Methodol 24(3):14
Scanniello G, Gravino C, Tortora G, Genero M, Risi M, Cruz-Lemus JA, Dodero G (2015) Studying the effect of uml-based models on source-code comprehensibility: Results from a long-term investigation. In: Springer (ed.) Proceedings of International Conference on Product-Focused Software Process Improvement, vol 9459. Lecture Notes in Computer Science, pp 311–327
Settimi R, Cleland-Huang J, Khadra OB, Mody J, Lukasik W, DePalma C (2004) Supporting software evolution through dynamically retrieving traces to uml artifacts. In: Proceedings of International Workshop on Principles of Software Evolution, pp 49–54
Shull F, Carver JC, Vegas S, Juzgado NJ (2008) The role of replications in empirical software engineering. Empir Softw Eng 13(2):211–218
Sillito J, Murphy GC, De Volder K (2008) Asking and answering questions during a programming change task. IEEE Trans Softw Eng 34(4):434–451
Tang A, Jin Y, Han J (2007) A rationale-based architecture model for design traceability and reasoning. J Syst Softw 80(6):918–934
Tang A, Nicholson A, Jin Y, Han J (2007) Using bayesian belief networks for change impact analysis in architecture design. J Syst Softw 80(1):127–148
Wohlin C, Runeson P, Höst M., Ohlsson M, Regnell B, Wesslén A (2012) Experimentation in software engineering. Springer, Berlin
Zhi J, Sun B, Garousi G, Shahnewaz SM, Ruhe G (2015) Cost, benefits and quality of software development documentation: A systematic mapping. J Syst Softw 99:175–198
Acknowledgements
The authors would like to thank the participants in the experiments and all the people who supported the research presented in this paper. This work has been partially supported by the SEQUOIA Project, (TIN2015-63502-C3-1-R) (MINECO/FEDER) funded by Fondo Europeo de Desarrollo Regional and Ministerio de Econom/ y Competitividad
Author information
Authors and Affiliations
Corresponding author
Additional information
Communicated by: Richard Paige
Rights and permissions
About this article
Cite this article
Scanniello, G., Gravino, C., Genero, M. et al. Do software models based on the UML aid in source-code comprehensibility? Aggregating evidence from 12 controlled experiments. Empir Software Eng 23, 2695–2733 (2018). https://doi.org/10.1007/s10664-017-9591-4
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10664-017-9591-4