Empirical Software Engineering

, Volume 21, Issue 1, pp 212–259 | Cite as

Does the level of detail of UML diagrams affect the maintainability of source code?: a family of experiments

  • Ana M. Fernández-SáezEmail author
  • Marcela Genero
  • Danilo Caivano
  • Michel R. V. Chaudron


Although the UML is considered to be the de facto standard notation with which to model software, there is still resistance to model-based development. UML modeling is perceived to be expensive and not necessarily cost-effective. It is therefore important to collect empirical evidence concerning the conditions under which the use of UML makes a practical difference. The focus of this paper is to investigate whether and how the Level of Detail (LoD) of UML diagrams impacts on the performance of maintenance tasks in a model-centric approach. A family of experiments consisting of one controlled experiment and three replications has therefore been carried out with 81 students with different abilities and levels of experience from 3 countries (The Netherlands, Spain, and Italy). The analysis of the results of the experiments indicates that there is no strong statistical evidence as to the influence of different LoDs. The analysis suggests a slight tendency toward better results when using low LoD UML diagrams, especially if used for the modification of the source code, while a high LoD would appear to be helpful in understanding the system. The participants in our study also favored low LoD diagrams because they were perceived as easier to read. Although the participants expressed a preference for low LoD diagrams, no statistically significant conclusions can be drawn from the set of experiments. One important finding attained from this family of experiments was that the participants minimized or avoided the use of UML diagrams, regardless of their LoD. This effect was probably the result of using small software systems from well-known domains as experimental materials.


UML diagrams Software maintenance Level of detail Controlled experiment Replication Family of experiments 



This work has been funded by the following projects: GEODAS-BC (Ministerio de Economía y Competitividad and Fondo Europeo de Desarrollo Regional FEDER, TIN2012-37493-C03-01) and IMPACTUM (Consejería de Educación, Ciencia y Cultura de la Junta de Comunidades de Castilla La Mancha, y Fondo Europeo de Desarrollo Regional FEDER, PEII11-0330-4414).


  1. Abrial JR (1996) The B-Book. Cambridge University Press, New YorkCrossRefzbMATHGoogle Scholar
  2. Anda B, Hansen K, Gullesen I, Thorsen HK (2006) Experiences from introducing UML-based development in a large safety-critical project. Empir Softw Eng 11:555–581CrossRefGoogle Scholar
  3. Arisholm E, Briand LC, Hove SE, Labiche Y (2006) The impact of UML documentation on software maintenance: an experimental evaluation. IEEE Trans Softw Eng 32:365–381CrossRefGoogle Scholar
  4. Basili V, Shull F, Lanubile F (1999) Building knowledge through families of experiments. IEEE Trans Softw Eng 25:456–473CrossRefGoogle Scholar
  5. Basili V, Weiss D (1984) A methodology for collecting valid software engineering data. IEEE Trans Softw Eng 10:728–738CrossRefGoogle Scholar
  6. Biostat (2006) Comprehensive Meta-Analysis v2. Biostat, EnglewoodGoogle Scholar
  7. Briand LC, Labiche Y, Di Penta M, Yan-Bondoc H (2005) An experimental investigation of formality in UML-based development. IEEE Trans Softw Eng 31:833–849CrossRefGoogle Scholar
  8. Briand LC, Wüst J, Lounis H (2001) Replicated Case Studies for Investigating Quality Factors in Object-Oriented Designs. Empir Softw Eng 6:11–58. doi: 10.1023/A:1009815306478 CrossRefzbMATHGoogle Scholar
  9. Bruegge B, Dutoit AH (2010) Object-oriented software engineering: using UML, patterns, and Java. Prentice Hall, BostonGoogle Scholar
  10. Cant S, Jeffery D, Henderson-Sellers B (1995) A conceptual model of cognitive complexity of elements of the programming process. Inf Softw Technol 37:351–362. doi: 10.1016/0950-5849(95)91491-H CrossRefGoogle Scholar
  11. Carver J (2010) Towards Reporting Guidelines for Experimental Replications: A Proposal. Proceedings of the 1st International Workshop on Replication in Empirical Software Engineering Research (RESER) [Held during ICSE 2010]Google Scholar
  12. Cohen D, Lindvall M, Costa P (2004) An introduction to agile methods. Adv Comput 62:2–67Google Scholar
  13. Conover WJ (1998) Practical Nonparametric Statistics, 3rd ed. WileyGoogle Scholar
  14. Cruz-Lemus JA, Genero M, Caivano D et al (2010) Assessing the influence of stereotypes on the comprehension of UML sequence diagrams: A family of experiments. Inf Softw Technol 53:1391–1403CrossRefGoogle Scholar
  15. Devore JL, Farnum N (1999) Applied Statistics for Engineers and Scientists. Duxbury Press, NYGoogle Scholar
  16. Dobing B, Parsons J (2006) How UML is used? Commun ACM 49:109–113CrossRefGoogle Scholar
  17. Dzidek WJ, Arisholm E, Briand LC (2008) A realistic empirical evaluation of the costs and benefits of UML in software maintenance. IEEE Trans Softw Eng 34:407–432CrossRefGoogle Scholar
  18. Erickson J, Siau K (2007) Theoretical and practical complexity of modeling methods. Commun ACM 50:46–51CrossRefGoogle Scholar
  19. Ericksson HE, Penker M, Lyons B, Fado D (2004) UML 2 Toolkit. WileyGoogle Scholar
  20. Fernández-Sáez AM, Chaudron MRV, Genero M, Ramos I (2013a) Are forward designed or reverse-engineered UML diagrams more helpful for code maintenance?: a controlled experiment. Proceedings of the 17th International Conference on Evaluation and Assessment in Software Engineering. ACM, New York, pp 60–71Google Scholar
  21. Fernández-Sáez AM, Genero M, Chaudron MRV (2013b) Empirical studies concerning the maintenance of UML diagrams and their use in the maintenance of code: A systematic mapping study. Inf Softw Technol 55:1119–1142CrossRefGoogle Scholar
  22. Fernández-Sáez AM, Genero M, Chaudron MRV et al (2014) Are Forward Designed or Reverse-Engineered UML diagrams more helpful for code maintenance?: A family of experiments. Inf Softw Technol. doi: 10.1016/j.infsof.2014.05.014 Google Scholar
  23. Fernández-Sáez AM, Genero M, Chaudron MRV (2012) Does the Level of Detail of UML Models Affect the Maintainability of Source Code? Proceedings of the Experiences and Empirical Studies in Software Modelling Workshop (EESSMod’11) at MODELS 2011. LNCS 7167, Wellington, New Zealand, pp 133–147Google Scholar
  24. Fjeldstad RK, Hamlen WT (1979) Application Program Maintenance Study: Report to Our Respondents. Proceedings of GUIDE 48Google Scholar
  25. Garousi G, Garousi V, Moussavi M, et al. (2013) Evaluating Usage and Quality of Technical Software Documentation: An Empirical Study. Proceedings of the 17th International Conference on Evaluation and Assessment in Software Engineering (EASE’2013). ACM, pp 24–35Google Scholar
  26. Glass R (2002) Facts and fallacies of software engineering. Addison-WesleyGoogle Scholar
  27. Glässer U, Gotzhein R, Prinz A (2003) The formal semantics of SDL-2000: Status and perspectives. Comput Netw 42:343–358. doi: 10.1016/S1389-1286(03)00247-0 CrossRefzbMATHGoogle Scholar
  28. Gravino C, Tortora G, Scanniello G (2010) An empirical investigation on the relation between analysis models and source code comprehension. Proceedings of the 2010 ACM Symposium on Applied Computing (SAC’2010). ACM, pp 2365–2366Google Scholar
  29. Grossman M, Aronson JE, McCarthy RV (2005) Does UML make the grade? Insights from the software development community. Inf Softw Technol 47:383–397CrossRefGoogle Scholar
  30. Hannay JE, Dybå T, Arisholm E, Sjøberg DIK (2009) The effectiveness of pair programming: A meta-analysis. Inf Softw Technol 51:1110–1122CrossRefGoogle Scholar
  31. Hedges LV, Olkin I (1985) Statistical Methods for Meta-Analysis. Academia Press, New YorkzbMATHGoogle Scholar
  32. Höst M, Regnell B, Wholin C (2000) Using students as subjects - a comparative study of students and professionals in lead-time impact assessment. Proceedings of the 4th Conference on Empirical Assessment and Evaluation in Software Engineering. pp 201–214Google Scholar
  33. ISO/IEC (2014) ISO/IEC 25001: Systems and software engineering - Systems and software Quality Requirements and Evaluation (SQuaRE) - Planning and management. International Organization for StandarizationGoogle Scholar
  34. Jedlitschka A, Ciolkowoski M, Pfahl D (2008) Reporting experiments in software engineering. Guide to Advanced Empirical Software EngineeringGoogle Scholar
  35. Juristo N, Moreno A (2001) Basics of software engineering experimentation. Kluwer Academic Publishers, BostonCrossRefzbMATHGoogle Scholar
  36. Juristo N, Vegas S, Solari M et al (2013) A process for managing interaction between experimenters to get useful similar replications. Inf Softw Technol 55:215–225CrossRefGoogle Scholar
  37. Kampenes V, Dybå T, Hannay JE, Sjoberg DIK (2007) A Systematic Review of Effect Size in Software Engineering Experiments. Inf Softw Technol 49:1073–1086CrossRefGoogle Scholar
  38. Karahasanovic A, Thomas R (2007) Difficulties experienced by students in maintaining object-oriented Systems: an empirical study. Proceedings of the Australasian Computing Education Conference (ACE’2007). pp 81–87Google Scholar
  39. Kirk RE (1995) Experimental design. procedures for the behavioural sciences. Brooks/Cole Publishing Company, BelmontGoogle Scholar
  40. Kitchenham BA, Pfleeger S, Hoaglin DC et al (2002) Preliminary Guidelines for Empirical Research in Software Engineering. IEEE Trans Softw Eng 28:721–734CrossRefGoogle Scholar
  41. Lange CFJ, Chaudron MRV, Muskens J (2006) In practice: UML software architecture and design description. IEEE Softw 23:40–46CrossRefGoogle Scholar
  42. Lauesen S (2002) Software Requirements: Styles and Techniques. Addison-Wesley, UKGoogle Scholar
  43. Lientz BP, Swanson EB (1980) Software Maintenance Management. Addison -Wesley, MassachusettsGoogle Scholar
  44. Lindsay RM, Ehrenberg A (1993) The design of replicated studies. Am Stat 47:217–228Google Scholar
  45. Nugroho A (2009) Level of detail in UML models and its impact on model comprehension: A controlled experiment. Inf Softw Technol 51:1670–1685CrossRefGoogle Scholar
  46. Nugroho A, Chaudron MRV (2009) Evaluating the impact of UML modeling on software quality: An industrial case study. LNCS 5795:181–195Google Scholar
  47. Nugroho A, Chaudron MRV (2008) A survey into the rigor of UML use and its perceived impact on quality and productivity. Proceedings of the Second ACM-IEEE international symposium on Empirical software engineering and measurement (ESEM 2008). ACM, New York, pp 90–99Google Scholar
  48. OMG (2010) The Unified Modeling Language. Documents associated with UML version 2.3.
  49. Oppenheim AN (2000) Questionnaire Design, Interviewing and Attitude Measurement, 0002-New ed. Bloomsbury Academic, United KingdomGoogle Scholar
  50. Pippenger N (1978) Complexity Theory. Scientific American 238:Google Scholar
  51. Poels G, Dedene G (2000) Measures for Assessing Dynamic Complexity Aspects of Object-oriented Conceptual Schemes. Proceedings of the 19th International Conference on Conceptual Modeling. Springer, Berlin, pp 499–512Google Scholar
  52. Pressman RS (2005) Software engineering: a practitioners approach, seventh. McGraw Hill, New YorkGoogle Scholar
  53. Roehm T, Tiarks R, Koschke R, Maalej W (2012) How Do Professional Developers Comprehend Software? Proceedings of the 34th International Conference on Software Engineering. IEEE Press, Piscataway, pp 255–265Google Scholar
  54. Scanniello G, Gravino C, Genero M, et al. (2013) On the Impact of UML Analysis Models on Source Code Comprehensibility and Modifiability. ACM Transactions On Software Engineering And Methodology (In press) 26Google Scholar
  55. Scanniello G, Gravino C, Tortora G (2012) Does the Combined use of Class and Sequence Diagrams Improve the Source Code Comprehension? Results from a Controlled Experiment. Proceedings of the Experiences and Empirical Studies in Software Modelling Workshop (EESSMoD´2012)Google Scholar
  56. Scanniello G, Gravino C, Tortora G (2010) Investigating the Role of UML in the Software Modeling and Maintenance - A Preliminary Industrial Survey. Proceedings of the 12th International Conference on Enterprise Information Systems (ICEIS’2010). Funchal, Madeira, Portugal, pp 141–148 Google Scholar
  57. Sheskin D (2007) Handbook of Parametric and Nonparametric Statistical Procedures, 4th edn. Chapman and Hall, New YorkzbMATHGoogle Scholar
  58. Siau K (1999) Information Modeling and Method Engineering: A Psychological Perspective. J Database Manag 10:44–50CrossRefGoogle Scholar
  59. Singer J, Vinson NG (2002) Ethical Issues in Empirical Studies of Software Engineering. IEEE Trans Softw Eng 28:1171–1180. doi: 10.1109/TSE.2002.1158289 CrossRefGoogle Scholar
  60. Sjøberg DIK, Hannay JE, Hansen O et al (2005) A survey of controlled experiments in software engineering. IEEE Trans Softw Eng 31:733–753CrossRefGoogle Scholar
  61. Spivey JM (1989) The Z Notation: A Reference Manual. Prentice-Hall, New YorkzbMATHGoogle Scholar
  62. SPSS (2003) SPSS 12.0, syntax reference guide. SPSS Inc, ChicagoGoogle Scholar
  63. Tryggeseth E (1997) Report from an Experiment: Impact of Documentation on Maintenance. J Empir Softw Eng 2:201–207CrossRefGoogle Scholar
  64. Vegas S, Juristo N, Moreno A, et al. (2006) Analysis of the influence of communication between researchers on experiment replication. Proceedings of the ACM/IEEE international symposium on Empirical software engineering (ISESE’2006). pp 28–37Google Scholar
  65. Verelst J (2004) The influence of the level of abstraction on the evolvability of conceptual models of information systems. Proceedings of the International Symposium on Empirical Software Engineering (ISESE’04). pp 17–26Google Scholar
  66. Wieringa RJ (2003) Chapter 8 - Entity-Relationship Diagrams. In: Wieringa RJ (ed) Design Methods for Reactive Systems. Morgan Kaufmann, San Francisco, pp 77–88CrossRefGoogle Scholar
  67. Wohlin C, Runeson P, Host M et al (1999) Experimentation in Software Engineering: An Introduction. Kluwer Academic Publishers, BostonGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2014

Authors and Affiliations

  • Ana M. Fernández-Sáez
    • 1
    • 5
    Email author
  • Marcela Genero
    • 1
  • Danilo Caivano
    • 2
    • 3
  • Michel R. V. Chaudron
    • 4
    • 5
  1. 1.ALARCOS Research Group, Department of Technologies and Information SystemsUniversity of Castilla-La ManchaCiudad RealSpain
  2. 2.Department of InformaticsUniversity of BariBariItaly
  3. 3.SER&Practices s.r.lSpin Off Company of the University of BariBariItaly
  4. 4.Joint Computer Science and Engineering Department of ChalmersUniversity of Technology and University of GothenburgGö̈teborgSweden
  5. 5.Leiden Institute of Advanced Computer SciencesLeiden UniversityLeidenThe Netherlands

Personalised recommendations