Empirical Software Engineering

, Volume 23, Issue 3, pp 1743–1790 | Cite as

On the impact of state-based model-driven development on maintainability: a family of experiments using UniMod

  • Filippo Ricca
  • Marco Torchiano
  • Maurizio Leotta
  • Alessandro Tiso
  • Giovanna Guerrini
  • Gianna Reggio


Context: Model-driven approaches are well-known in the academia but one possible showstopper to a wider adoption in the industry is the limited empirical evidence for their claimed advantages and benefits, that could convince the decision makers. Objective: The aim of this work is gauging one of the claimed benefits of model-driven approaches, namely improvement in maintainability, with respect to a code-centric approach. Method: We conducted a family of five experiments involving 100 students that possessed different levels of education (64 Bachelor, 25 Master, and 11 PhD students; in groups sized 11 to 26 per individual experiment). In these experiments, UniMod – a State-based tool for Model-Driven Development using the Unified Modeling Language – is compared with Java-based code-centric programming, in a software maintenance scenario, with the goal of analyzing the effect on the time to perform the maintenance tasks, the correctness of the modified artifacts, and the efficiency. Results: The results show a reduction in time to accomplish the tasks and no impact on correctness. The adoption of the UniMod-MDD approach almost doubles the developers’efficiency, and in presence of a higher software engineering experience the efficiency is even three times higher. Conclusions: We found that the usage of the UniMod-MDD approach in a software maintenance scenario provides benefits over a pure code-centric approach. The benefits deriving from the UniMod-MDD approach are appreciable for all the categories of students, although with differences.


Model-driven development MDD UML Maintainability UniMod Family of experiments Replication and maintenance task 


  1. Abdi H (2007) Bonferroni and Sidak corrections for multiple comparisons. Sage, Thousand OaksGoogle Scholar
  2. Agner LTW, Soares IW, Stadzisz PC, Simão JM (2013) A Brazilian survey on UML and model-driven practices for embedded software development. J Syst Softw 86(4):997–1005CrossRefGoogle Scholar
  3. Arisholm E, Briand L, Hove S, Labiche Y (2006) The impact of UML documentation on software maintenance: an experimental evaluation. IEEE Trans Softw Eng 32(6):365–381CrossRefGoogle Scholar
  4. Anda B, Hansen K, Gullesen I, Thorsen HK (2006) Experiences from introducing UML-based development in a large safety-critical project. Empir Softw Eng 11(4):555–581. CrossRefGoogle Scholar
  5. Baker R (1995) Modern permutation test software. In: Edgington E (ed) Randomization Tests, Marcel DeckerGoogle Scholar
  6. Ceccato M, Penta M, Falcarin P, Ricca F, Torchiano M, Tonella P (2014) A family of experiments to assess the effectiveness and efficiency of source code obfuscation techniques. Empirx Softw Eng 19(4):1040–1074. [Online]. Available: Google Scholar
  7. Ceri S, Fraternali P, Bongio A, Brambilla M, Comai S, Matera M (2002) Designing data-intensive web applications. Morgan Kaufmann Publishers Inc., San FranciscoGoogle Scholar
  8. Chapin N, Hale JE, Kham KM, Ramil JF, Tan W-G (2001) Types of software evolution and software maintenance. J Softw Maint 13(1):3–30. [Online]. Available: CrossRefzbMATHGoogle Scholar
  9. Dzidek WJ, Arisholm E, Briand LC (2008) A realistic empirical evaluation of the costs and benefits of UML in software maintenance. IEEE Trans Softw Eng 34 (3):407–432. CrossRefGoogle Scholar
  10. Feitelson DG (2015) Using students as experimental subjects in software engineering research - A review and discussion of the evidence, CoRR, vol. abs/1512.08409. [Online]. Available: 1512.08409
  11. Fowler M (2003) UML distilled: a brief guide to the standard object modeling language, 3rd edn. Addison-Wesley Longman Publishing Co., Inc., BostonGoogle Scholar
  12. Grissom RJ, Kim JJ (2005) Effect sizes for research: A broad practical approach, 2nd edn. Lawrence Earlbaum Associates, New JerseyGoogle Scholar
  13. Gurov V, Mazin M, Narvsky A, Shalyto A (2007) Tools for support of automata-based programming. Programm Comput Softw 33:343–355CrossRefzbMATHGoogle Scholar
  14. Hamon G (2005) A denotational semantics for stateflow. In: Proceedings of the 5th ACM International Conference on Embedded Software, ser. EMSOFT ’05. ACM, New York, pp 164–172. [Online]. Available:
  15. Hamon G, Rushby J (2007) An operational semantics for stateflow. Int J Softw Tools Technol Transf 9(5):447–456. [Online]. Available: CrossRefzbMATHGoogle Scholar
  16. Harel D (1987) Statecharts: a visual formalism for complex systems. S Comput Programm 8(3):231–274. [Online]. Available: MathSciNetCrossRefzbMATHGoogle Scholar
  17. Heiberger R, Robbins N (2014) Design of diverging stacked bar charts for likert scales and other applications. J Stat Softw 57(5):1–32, 4. [Online]. Available: CrossRefGoogle Scholar
  18. Hutchinson J., Whittle J, Rouncefield M, Kristoffersen S (2011) Empirical assessment of MDE in industry. In: Proceedings of 33rd International Conference on Software Engineering (ICSE 2011). ACM, New York, pp 471–480Google Scholar
  19. Hovsepyan A, Scandariato R, Van Baelen S, Joosen W, Demeyer S (2011) Preserving aspects via automation: A maintainability study. In: Proceedings of 5th International Symposium on Empirical Software Engineering and Measurement (ESEM 2011). IEEE CS, Washington, pp 315–324. [Online]. Available:
  20. Jedlitschka A, Ciolkowski M, Pfahl D (2008) Reporting experiments in software engineering. In: Guide to Advanced Empirical Software Engineering. Springer, BerlinGoogle Scholar
  21. Kapteijns T, Jansen S, Brinkkemper S, Houet H, Barendse R (2009) A comparative case study of model driven development vs traditional development: the tortoise or the hare. In: 4th European Workshop on “From code centric to model centric software engineering: Practices, Implications and ROI” (C2M 2009). CTIT Workshop Proceedings Series, pp 22–33Google Scholar
  22. Kitchenham B, Al-Khilidar H, Babar M, Berry M, Cox K, Keung J, Kurniawati F, Staples M, Zhang H, Zhu L (2008) Evaluating guidelines for reporting empirical software engineering studies. Empir Softw Eng 13:97–121CrossRefGoogle Scholar
  23. Kleppe AG, Warmer J, Bast W (2003) MDA explained: the model driven architecture: practice and promise. Addison-Wesley Longman Publishing Co. Inc, BostonGoogle Scholar
  24. Leotta M, Ricca F, Antoniol G, Garousi V, Zhi J, Ruhe G (2013) A pilot experiment to quantify the effect of documentation accuracy on maintenance tasks. In: Proceedings of 29th International Conference on Software Maintenance (ICSM 2013). IEEE, pp 428–431.
  25. Liebel G, Marko N, Tichy M, Leitner A, Hansson J (2014) Assessing the state-of-practice of model-based engineering in the embedded systems domain. In: Dingel J, Schulte W, Ramos I, Abrahão S, Insfran E (eds) Proceedings of 17th International Conference on Model Driven Engineering Languages and Systems (MODELS 2014), ser. Lecture Notes in Computer Science, vol 8767. Springer, Berlin, pp 166–182. [Online]. Available:
  26. Martínez Y, Cachero C, Matera M, Abrahao S, Luján S (2011) Impact of MDE approaches on the maintainability of web applications: an experimental evaluation. In: Proceedings of 30th International Conference on Conceptual Modeling (ER 2011). Springer-Verlag, Berlin, pp 233–246. [Online]. Available:
  27. Martínez Y, Cachero C, Meliá S (2014) Empirical study on the maintainability of web applications: Model-driven engineering vs code-centric. Empir Softw Eng 19 (6):1887–1920. [Online]. Available: CrossRefGoogle Scholar
  28. Maraninchi F, Rémond Y (2001) Argos: an automaton-based synchronous language. Comput Lang 27(1–3):61–92. visual Formal Methods-VFM’99 Symposium. [Online]. Available: CrossRefzbMATHGoogle Scholar
  29. Mellegård N, Ferwerda A, Lind K, Heldal R, Chaudron MRV (2016) Impact of introducing domain-specific modelling in software maintenance: An industrial case study. IEEE Trans Softw Eng 42(3):245–260CrossRefGoogle Scholar
  30. Melia S, Gomez J, Perez S, Diaz O (2008) A model-driven development for gwt-based rich internet applications with ooh4ria. In: Proceedings of 8th International Conference on Web Engineering (ICWE 2008), pp 13–23Google Scholar
  31. Motulsky H (2010) Intuitive biostatistics: a non-mathematical guide to statistical thinking. Oxford University Press, Oxford. [Online]. Available: Google Scholar
  32. Mussbacher G, Amyot D, Breu R, Bruel J-M, Cheng B, Collet P, Combemale B, France R, Heldal R, Hill J, Kienzle J, Schöttle M, Steimann F, Stikkolorum D, Whittle J (2014) The relevance of model-driven engineering thirty years from now. In: Dingel J., Schulte W., Ramos I., Abrahão S, Insfran E (eds) Proceedings of 17th International Conference on Model Driven Engineering Languages and Systems (MODELS 2014), ser. Lecture Notes in Computer Science, vol 8767. Springer, Berlin, pp 183–200. [Online]. Available:
  33. Oppenheim AN (1992) Questionnaire Design, Interviewing and Attitude Measurement. Pinter, LondonGoogle Scholar
  34. Papotti P, do Prado A, de Souza W, Cirilo C, Pires L (2013) A quantitative analysis of model-driven code generation through software experimentation. In: Salinesi C, Norrie M, Pastor Ó (eds) Advanced Information Systems Engineering, ser. Lecture Notes in Computer Science, vol 7908. Springer, Berlin Heidelberg, pp 321–337. [Online]. Available:
  35. Reggio G, Leotta M, Ricca F (2014) Who knows/uses what of the UML: A personal opinion survey. In: Dingel J., Schulte W, Ramos I., Abrahão S, Insfran E (eds) Proceedings of 17th International Conference on Model Driven Engineering Languages and Systems (MODELS 2014), ser. Lecture Notes in Computer Science, vol 8767. Springer, Berlin, pp 149–165.
  36. Reggio G, Leotta M, Ricca F, Clerissi D (2015) What are the used UML diagram constructs? A document and tool analysis study covering activity and use case diagrams. In: Hammoudi S, Pires LF, Filipe J, das Neves RC (eds) Model-Driven Engineering and Software Development, ser. Communications in Computer and Information Science, vol 506. Springer, Berlin, pp 66–83. [Online]. Available:
  37. Pfleeger SL, Menezes W (2000) Marketing technology to software practitioners. IEEE Softw 17(1):27–33CrossRefGoogle Scholar
  38. Ricca F, Di Penta M, Torchiano M, Tonella P, Ceccato M, Visaggio CA (2008) Are Fit tables really talking?: a series of experiments to understand whether fit tables are useful during evolution tasks. In: Proceedings of 30th International Conference on Software Engineering (ICSE 2008). ACM, New York, pp 361–370. [Online]. Available:
  39. Ricca F, Di Penta M, Torchiano M, Tonella P, Ceccato M (2010) How developers’ experience and ability influence web application comprehension tasks supported by UML stereotypes: A series of four experiments. IEEE Trans Softw Eng 36:96–118CrossRefGoogle Scholar
  40. Ricca F, Leotta M, Reggio G, Tiso A, Guerrini G, Torchiano M (2012) Using UniMod for maintenance tasks: An experimental assessment in the context of model driven development. In: Proceedings of 4th International Workshop on Modeling in Software Engineering (MiSE 2012). IEEE, pp 77–83. [Online]. Available:
  41. R Core Team R (2013) A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna. ISBN 3-900051-07-0. [Online]. Available: Google Scholar
  42. Sheskin D (2007) Handbook of parametric and nonparametric statistical procedures, 4th edn. Chapman & All, UKzbMATHGoogle Scholar
  43. Shalyto AA, Tukkel NI (2001) SWITCH technology: an automated approach to developing software for reactive systems. Programm Comput Softw 27:260–276CrossRefzbMATHGoogle Scholar
  44. Shapiro SS, Wilk MB (1965) An analysis of variance test for normality (complete samples). Biometrika 52:3MathSciNetCrossRefzbMATHGoogle Scholar
  45. Sommerville I (2006) Software engineering. Addison-Wesley Longman Publishing Co. Inc, BostonzbMATHGoogle Scholar
  46. Stahl T, Voelter M, Czarnecki K (2006) Model-driven software development: technology, engineering, management. Wiley, New JerseyGoogle Scholar
  47. Svahnberg M, Aurum A, Wohlin C (2008) Using students as subjects-an empirical evaluation. In: Proceedings of 2nd International Symposium on Empirical Software Engineering and Measurement (ESEM 2008), pp 288–290Google Scholar
  48. Torchiano M, Tomassetti FCA, Ricca F, Tiso A, Reggio G (2013) Relevance, benefits, and problems of software modelling and model driven techniques–a survey in the Italian industry. J Syst Softw 86(8):2110–2126CrossRefGoogle Scholar
  49. Torchiano M (2015) effsize: Efficient effect size computation. r package version 0.5.5. [Online]. Available:
  50. Torchiano M, Scanniello G, Ricca F, Reggio G, Leotta M (2017) Do UML object diagrams affect design comprehensibility? Results from a family of four controlled experiments. J Vis Lang Comput 41:10–21. [Online]. Available: CrossRefGoogle Scholar
  51. Wheeler B (2016) lmPerm: Permutation tests for linear models. R package version 2.0. [Online]. Available:
  52. Whittle J, Hutchinson J, Rouncefield M (2014) The state of practice in model-driven engineering. IEEE Softw 31(3):79–85CrossRefGoogle Scholar
  53. Wohlin C, Runeson P, Höst M, Ohlsson M, Regnell B, Wesslén A (2012) Experimentation in software engineering. Springer-Verlag, Berlin HeidelbergCrossRefzbMATHGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2017

Authors and Affiliations

  1. 1.DIBRISUniversità di GenovaGenovaItaly
  2. 2.DAUINPolitecnico di TorinoTorinoItaly

Personalised recommendations