Empirical Software Engineering

, Volume 23, Issue 5, pp 2695–2733 | Cite as

Do software models based on the UML aid in source-code comprehensibility? Aggregating evidence from 12 controlled experiments

  • Giuseppe Scanniello
  • Carmine GravinoEmail author
  • Marcela Genero
  • José A. Cruz-Lemus
  • Genoveffa Tortora
  • Michele Risi
  • Gabriella Dodero


In this paper, we present the results of long-term research conducted in order to study the contribution made by software models based on the Unified Modeling Language (UML) to the comprehensibility of Java source-code deprived of comments. We have conducted 12 controlled experiments in different experimental contexts and on different sites with participants with different levels of expertise (i.e., Bachelor’s, Master’s, and PhD students and software practitioners from Italy and Spain). A total of 333 observations were obtained from these experiments. The UML models in our experiments were those produced in the analysis and design phases. The models produced in the analysis phase were created with the objective of abstracting the environment in which the software will work (i.e., the problem domain), while those produced in the design phase were created with the goal of abstracting implementation aspects of the software (i.e., the solution/application domain). Source-code comprehensibility was assessed with regard to correctness of understanding, time taken to accomplish the comprehension tasks, and efficiency as regards accomplishing those tasks. In order to study the global effect of UML models on source-code comprehensibility, we aggregated results from the individual experiments using a meta-analysis. We made every effort to account for the heterogeneity of our experiments when aggregating the results obtained from them. The overall results suggest that the use of UML models affects the comprehensibility of source-code, when it is deprived of comments. Indeed, models produced in the analysis phase might reduce source-code comprehensibility, while increasing the time taken to complete comprehension tasks. That is, browsing source code and this kind of models together negatively impacts on the time taken to complete comprehension tasks without having a positive effect on the comprehensibility of source code. One plausible justification for this is that the UML models produced in the analysis phase focus on the problem domain. That is, models produced in the analysis phase say nothing about source code and there should be no expectation that they would, in any way, be beneficial to comprehensibility. On the other hand, UML models produced in the design phase improve source-code comprehensibility. One possible justification for this result is that models produced in the design phase are more focused on implementation details. Therefore, although the participants had more material to read and browse, this additional effort was paid back in the form of an improved comprehension of source code.


Aggregation Heterogeneity Unified modeling language Controlled experiments 



The authors would like to thank the participants in the experiments and all the people who supported the research presented in this paper. This work has been partially supported by the SEQUOIA Project, (TIN2015-63502-C3-1-R) (MINECO/FEDER) funded by Fondo Europeo de Desarrollo Regional and Ministerio de Econom/ y Competitividad


  1. Abrahão SM, Gravino C, Pelozo EI, Scanniello G, Tortora G (2013) Assessing the effectiveness of sequence diagrams in the comprehension of functional requirements: Results from a family of five experiments. IEEE Trans Softw Eng 39 (3):327–342CrossRefGoogle Scholar
  2. Agarwal R, Sinha AP (2003) Object-oriented modeling with UML: a study of developers’ perceptions. Commun ACM 46(9):248–256CrossRefGoogle Scholar
  3. Anda B, Hansen K, Gullesen I, Thorsen HK (2006) Experiences from introducing UML-based development in a large safety-critical project. Empir Softw Eng 11(4):555–581CrossRefGoogle Scholar
  4. Antoniol G, Canfora G, Casazza G, De Lucia A, Merlo E (2002) Recovering traceability links between code and documentation. IEEE Trans Softw Eng 28(10):970–983CrossRefGoogle Scholar
  5. Arisholm E, Briand LC, Hove SE, Labiche Y (2006) The impact of UML documentation on software maintenance: An experimental evaluation. IEEE Trans Softw Eng 32(6):365–381CrossRefGoogle Scholar
  6. Basili VR, Rombach HD (1988) The TAME project: Towards improvement-oriented software environments. IEEE Trans Softw Eng 14(6):758–773CrossRefGoogle Scholar
  7. Basili V, Shull F, Lanubile F (1999) Building knowledge through families of experiments. IEEE Trans Softw Eng 25(4):456–473CrossRefGoogle Scholar
  8. Bavota G, Canfora G, Di Penta M, Oliveto R, Panichella S (2013) An empirical investigation on documentation usage patterns in maintenance tasks. In: Proceedings of International Conference on Software Maintenance. IEEE Computer Society, pp 210–219Google Scholar
  9. Bruegge B, Dutoit AH (2003) Object-oriented software engineering: using UML, Patterns and Java, 2nd edn. Prentice-Hall, Upper Saddle RiverGoogle Scholar
  10. Budgen D, Burn AJ, Brereton OP, Kitchenham B, Pretorius R (2011) Empirical evidence about the UML: a systematic literature review. Softw Pract Exper 41(4):363–392CrossRefGoogle Scholar
  11. Cariou E, Beugnard A, Jezequel JM (2002) An architecture and a process for implementing distributed collaborations. In: Proceedings of International Enterprise Distributed Object Computing, pp 132–143Google Scholar
  12. Carver J, Jaccheri L, Morasca S, Shull F (2003) Issues in using students in empirical studies in software engineering education. In: Proceedings of International Symposium on Software Metrics. IEEE Computer Society, pp 239–250Google Scholar
  13. Corazza A, Maggio V, Scanniello G (2016) Coherence of comments and method implementations: a dataset and an empirical investigation. Softw Q J:1–27.
  14. Dzidek WJ, Arisholm E, Briand LC (2008) A realistic empirical evaluation of the costs and benefits of UML in software maintenance. IEEE Trans Softw Eng 34 (3):407–432CrossRefGoogle Scholar
  15. Eclipse Modeling Framework (EMF) (2012)
  16. Erickson J, Siau K (2007) Theoretical and practical complexity of modeling methods. Commun ACM 50(8):46–51CrossRefGoogle Scholar
  17. Fernȧndez-Sȧez AM, Chaudron MRV, Genero M (2013) Exploring costs and benefits of using UML on maintenance: Preliminary findings of a case study in a large IT department. In: Proceedings of the International Workshop on Experiences and Empirical Studies in Software Modeling co-located with the International Conference on Model Driven Engineering Languages and Systems, pp 33–42Google Scholar
  18. Fernȧndez-Sȧez AM, Caivano D, Genero M, Chaudron MRV (2015) On the use of UML documentation in software maintenance: Results from a survey in industry. In: Proceedings of ACM/IEEE International Conference on Model Driven Engineering Languages and Systems, pp 292–301Google Scholar
  19. Fernȧndez-Sȧez AM, Genero M, Caivano D, Chaudron MRV (2016) Does the level of detail of UML diagrams affect the maintainability of source code?: a family of experiments. Empir Softw Eng 21(1):212–259CrossRefGoogle Scholar
  20. Fernȧndez-Sȧez AM, Genero M, Chaudron MRV (2013) Empirical studies concerning the maintenance of UML diagrams and their use in the maintenance of code: A systematic mapping study. Inf Softw Technol 55(7):1119–1142CrossRefGoogle Scholar
  21. Fernȧndez-Sȧez AM, Genero M, Chaudron MRV, Caivano D, Ramos I (2015) Are forward designed or reverse-engineered UML diagrams more helpful for code maintenance?: A family of experiments. Inf Softw Technol 57:644–663CrossRefGoogle Scholar
  22. Fluri B, Wursch M, Gall H (2007) Do code and comments co-evolve? on the relation between source code and comment changes. In: Proceedings of the Working Conference on Reverse Engineering. IEEE Computer Society, pp 70–79Google Scholar
  23. Fu R, Gartlehner G, Grant M, Shamliyan T, Sedrakyan A, Wilt TJ, Griffith L, Oremus M, Raina P, Ismaila A, Santaguida P, Lau J, Trikalinos TA (2011) Conducting quantitative synthesis when comparing medical interventions: AHRQ and the effective health care program. J Clin Epidemiol 64(11):1187–1197CrossRefGoogle Scholar
  24. Gamma E, Helm R, Johnson R, Vlissides J (1995) Design patterns: elements of reusable object oriented software. Addison-Wesley, BostonzbMATHGoogle Scholar
  25. Garousi G, Garousi V, Moussavi M, Ruhe G, Smith B (2013) Evaluating usage and quality of technical software documentation: an empirical study. In: Proceedings of International Conference on Evaluation and Assessment in Software Engineering, pp 24–35Google Scholar
  26. Gravino C, Tortora G, Scanniello G (2010) An empirical investigation on the relation between analysis models and source code comprehension. In: Proceedings of the International Symposium on Applied Computing. ACM, pp 2365–2366Google Scholar
  27. Gravino C, Scanniello G, Tortora G (2015) Source-code comprehension tasks supported by UML design models: Results from a controlled experiment and a differentiated replication. J Vis Lang Comput 28:23–38CrossRefGoogle Scholar
  28. Grossman M, Aronson JE, McCarthy RV (2005) Does UML make the grade? Insights from the software development community. Inf Softw Technol 47(6):383–397CrossRefGoogle Scholar
  29. Guéhéneuc YG (2007) P-mart: Pattern-like micro architecture repository. In: Proceedings of EuroPLoP Focus Group on Pattern RepositoriesGoogle Scholar
  30. Hammad M, Collard ML, Maletic JI (2011) Automatically identifying changes that impact code-to-design traceability during evolution. Softw Qual J 19(1):35–64CrossRefGoogle Scholar
  31. Higgins JPT, Green S (2008) Cochrane handbook for systematic reviews of interventions, 5th edn. The Cochrane Collaboration, LondonCrossRefGoogle Scholar
  32. Huedo-Medina TB, Sánchez-Meca J, Marín-Martínez F, Botella J (2006) Assessing heterogeneity in meta-analysis: Q statistic or i2 index? Psychol Methods 11 (2):193–206CrossRefGoogle Scholar
  33. Hutchinson JE, Whittle J, Rouncefield M, Kristoffersen S (2011) Empirical assessment of MDE in industry. In: Proceedings of the International Conference on Software Engineering, pp 471–480Google Scholar
  34. ISO (1991) Information Technology–Software Product Evaluation: Quality Characteristics and Guidelines for their Use ISO/IEC IS 9126. ISO, GenevaGoogle Scholar
  35. ISO (2000) ISO 9241-11: Ergonomic requirements for office work with visual display terminals (VDTs) – Part 9: Requirements for non-keyboard input devices. ISO, GenevaGoogle Scholar
  36. ISO (2011) ISO/IEC 25010 Systems and software engineering – Systems and software Quality Requirements and Evaluation (SQuaRE) – System and software quality models. ISO, GenevaGoogle Scholar
  37. Jedlitschka A, Ciolkowski M, Pfahl D, Sjoberg D (2008) Reporting experiments in software engineering. In: Shull F, Singer J (eds) Guide to Advanced Empirical Software Engineering, Springer, pp 201–228Google Scholar
  38. Jiang ZM, Hassan AE (2006) Examining the evolution of code comments in postgresql. In: Proceedings of Mining Software Repositories. ACM, pp 179–180Google Scholar
  39. Juristo N, Moreno A (2001) Basics of software engineering experimentation. Kluwer Academic Publishers, DordrechtCrossRefzbMATHGoogle Scholar
  40. Kitchenham B, Pfleeger S, Pickard L, Jones P, Hoaglin D, El Emam K, Rosenberg J (2002) Preliminary guidelines for empirical research in software engineering. IEEE Trans Softw Eng 28(8):721–734CrossRefGoogle Scholar
  41. Kitchenham B, Charters S (2007) Guidelines for performing systematic literature reviews in software engineeringGoogle Scholar
  42. Lehnert S, Farooq Q u a, Riebisch M (2013) Rule-based impact analysis for heterogeneous software artifacts. In: Proceedings of the European Conference on Software Maintenance and Reengineering, pp 209–218Google Scholar
  43. Leotta M, Ricca F, Antoniol G, Garousi V, Zhi J, Ruhe G (2013) A pilot experiment to quantify the effect of documentation accuracy on maintenance tasks. In: Proceedings of International Conference on Software Maintenance, pp 428–431Google Scholar
  44. OMG (2005) Unified modeling language (UML) specification version 2.0. Technical report, Object Management GroupGoogle Scholar
  45. Oppenheim AN (1992) Questionnaire design, interviewing and attitude measurement, Pinter, LondonGoogle Scholar
  46. Pavalkis S, Nemuraite L (2013) Process for applying derived property based traceability framework in software and systems development life cycle. Springer, Berlin Heidelberg, pp 122–133Google Scholar
  47. Pavalkis S, Nemuraite L, Butkiene R (2013) Derived properties: A user friendly approach to improving model traceability. Inf Technol Control 42(1):48–60Google Scholar
  48. Pickard L, Kitchenham B, Jones P (1998) Combining empirical results in software engineering. Inf Softw Technol 40(14):811–821CrossRefGoogle Scholar
  49. Ried K (2008) Interpreting and understanding meta-analysis graphs - A practical guide, vol 35. Australian College of General Practitioners, East MelbourneGoogle Scholar
  50. Salviulo F, Scanniello G (2014) Dealing with identifiers and comments in source code comprehension and maintenance: Results from an ethnographically-informed study with students and professionals. In: Proceedings of International Conference on Evaluation and Assessment in Software Engineering. ACMGoogle Scholar
  51. Scanniello G, Gravino C, Risi M, Tortora G (2010) A controlled experiment for assessing the contribution of design pattern documentation on software maintenance. In: Proceedings of the Symposium on Empirical Software Engineering and Measurement. ACMGoogle Scholar
  52. Scanniello G, Gravino C, Tortora G (2010) Investigating the role of UML in the software modeling and maintenance - a preliminary industrial survey. In: Proceedings of International Conference on Enterprise Information Systems, pp 141–148Google Scholar
  53. Scanniello G, Gravino C, Genero M, Cruz-Lemus JA, Tortora G (2014) On the impact of UML analysis models on source code comprehensibility and modifiability. ACM Trans Softw Eng Methodol 23(2):13:1–13:26CrossRefGoogle Scholar
  54. Scanniello G, Gravino C, Risi M, Tortora G, Dodero G (2015) Documenting design-pattern instances: A family of experiments on source-code comprehensibility. ACM Trans Softw Eng Methodol 24(3):14CrossRefGoogle Scholar
  55. Scanniello G, Gravino C, Tortora G, Genero M, Risi M, Cruz-Lemus JA, Dodero G (2015) Studying the effect of uml-based models on source-code comprehensibility: Results from a long-term investigation. In: Springer (ed.) Proceedings of International Conference on Product-Focused Software Process Improvement, vol 9459. Lecture Notes in Computer Science, pp 311–327Google Scholar
  56. Settimi R, Cleland-Huang J, Khadra OB, Mody J, Lukasik W, DePalma C (2004) Supporting software evolution through dynamically retrieving traces to uml artifacts. In: Proceedings of International Workshop on Principles of Software Evolution, pp 49–54Google Scholar
  57. Shull F, Carver JC, Vegas S, Juzgado NJ (2008) The role of replications in empirical software engineering. Empir Softw Eng 13(2):211–218CrossRefGoogle Scholar
  58. Sillito J, Murphy GC, De Volder K (2008) Asking and answering questions during a programming change task. IEEE Trans Softw Eng 34(4):434–451CrossRefGoogle Scholar
  59. Tang A, Jin Y, Han J (2007) A rationale-based architecture model for design traceability and reasoning. J Syst Softw 80(6):918–934CrossRefGoogle Scholar
  60. Tang A, Nicholson A, Jin Y, Han J (2007) Using bayesian belief networks for change impact analysis in architecture design. J Syst Softw 80(1):127–148CrossRefGoogle Scholar
  61. Wohlin C, Runeson P, Höst M., Ohlsson M, Regnell B, Wesslén A (2012) Experimentation in software engineering. Springer, BerlinCrossRefzbMATHGoogle Scholar
  62. Zhi J, Sun B, Garousi G, Shahnewaz SM, Ruhe G (2015) Cost, benefits and quality of software development documentation: A systematic mapping. J Syst Softw 99:175–198CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  • Giuseppe Scanniello
    • 1
  • Carmine Gravino
    • 2
    Email author
  • Marcela Genero
    • 3
  • José A. Cruz-Lemus
    • 3
  • Genoveffa Tortora
    • 2
  • Michele Risi
    • 2
  • Gabriella Dodero
    • 4
  1. 1.University of BasilicataPotenzaItaly
  2. 2.University of SalernoFiscianoItaly
  3. 3.University of Castilla-La ManchaCiudad RealSpain
  4. 4.Free University of BozenBolzano-BozenItaly

Personalised recommendations