Skip to main content
SpringerLink
Log in

On the impact of state-based model-driven development on maintainability: a family of experiments using UniMod

  • Published:
Empirical Software Engineering Aims and scope Submit manuscript

Abstract

Context: Model-driven approaches are well-known in the academia but one possible showstopper to a wider adoption in the industry is the limited empirical evidence for their claimed advantages and benefits, that could convince the decision makers. Objective: The aim of this work is gauging one of the claimed benefits of model-driven approaches, namely improvement in maintainability, with respect to a code-centric approach. Method: We conducted a family of five experiments involving 100 students that possessed different levels of education (64 Bachelor, 25 Master, and 11 PhD students; in groups sized 11 to 26 per individual experiment). In these experiments, UniMod – a State-based tool for Model-Driven Development using the Unified Modeling Language – is compared with Java-based code-centric programming, in a software maintenance scenario, with the goal of analyzing the effect on the time to perform the maintenance tasks, the correctness of the modified artifacts, and the efficiency. Results: The results show a reduction in time to accomplish the tasks and no impact on correctness. The adoption of the UniMod-MDD approach almost doubles the developers’efficiency, and in presence of a higher software engineering experience the efficiency is even three times higher. Conclusions: We found that the usage of the UniMod-MDD approach in a software maintenance scenario provides benefits over a pure code-centric approach. The benefits deriving from the UniMod-MDD approach are appreciable for all the categories of students, although with differences.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16

Similar content being viewed by others

Notes

  1. http://www.andromda.org/

  2. http://www.mentor.com/products/sm/model_development/

  3. http://www.mathworks.com/products/stateflow/

  4. http://is.ifmo.ru/unimod-projects-en/svetofor/

  5. Here, for space reasons we will describe more in detail Svetofor than Telepay.

  6. http://unimod.sourceforge.net/

  7. This odd naming convention (e.g., o1.z14) is a peculiarity of UniMod to reduce the diagrams size.

  8. The complete documentation of Svetofor is available at: http://is.ifmo.ru/unimod-projects-en/svetofor/

  9. In our study we have not measured if this capability has contributed, or not, to the obtained results.

  10. http://sepl.dibris.unige.it/UniModVSJava.php

  11. http://is.ifmo.ru/unimod-projects-en/

  12. https://www.mathworks.com/help/stateflow/examples/modeling-an-intersection-of-two-1-way-streets-using-stateflow.html

  13. http://is.ifmo.ru/unimod-projects-en/teleplay/

  14. The number of test cases is different for each maintenance task.

  15. TotalCorrectness is divided by 4 since a fully correct task is assigned a score of 4, while TotalTime is divided by 60 because we want to measure tasks per hour while the time is measured in minutes.

  16. For both projects, we used the original documentation available at: http://is.ifmo.ru/unimod-projects-en/

  17. This is a simplified structure, since the variables are nominal they will be represented by n − 1indicator variables, where n is the number of levels.

  18. In some experiments of the family, the number of participants is different between the two treatments since some of them took part in a laboratory session only.

  19. 0.05 divided by the number of comparisons, i.e., 7.

  20. http://portofino.manydesigns.com/en

  21. http://www.mathworks.com/products/stateflow/

  22. http://www.mathworks.com/products/simulink-coder/

References

  • Abdi H (2007) Bonferroni and Sidak corrections for multiple comparisons. Sage, Thousand Oaks

    Google Scholar 

  • Agner LTW, Soares IW, Stadzisz PC, Simão JM (2013) A Brazilian survey on UML and model-driven practices for embedded software development. J Syst Softw 86(4):997–1005

    Article  Google Scholar 

  • Arisholm E, Briand L, Hove S, Labiche Y (2006) The impact of UML documentation on software maintenance: an experimental evaluation. IEEE Trans Softw Eng 32(6):365–381

    Article  Google Scholar 

  • Anda B, Hansen K, Gullesen I, Thorsen HK (2006) Experiences from introducing UML-based development in a large safety-critical project. Empir Softw Eng 11(4):555–581. https://doi.org/10.1007/s10664-006-9020-6

    Article  Google Scholar 

  • Baker R (1995) Modern permutation test software. In: Edgington E (ed) Randomization Tests, Marcel Decker

  • Ceccato M, Penta M, Falcarin P, Ricca F, Torchiano M, Tonella P (2014) A family of experiments to assess the effectiveness and efficiency of source code obfuscation techniques. Empirx Softw Eng 19(4):1040–1074. [Online]. Available: https://doi.org/10.1007/s10664-013-9248-x

    Google Scholar 

  • Ceri S, Fraternali P, Bongio A, Brambilla M, Comai S, Matera M (2002) Designing data-intensive web applications. Morgan Kaufmann Publishers Inc., San Francisco

    Google Scholar 

  • Chapin N, Hale JE, Kham KM, Ramil JF, Tan W-G (2001) Types of software evolution and software maintenance. J Softw Maint 13(1):3–30. [Online]. Available: http://dl.acm.org/citation.cfm?id=371697.371701

    Article  MATH  Google Scholar 

  • Dzidek WJ, Arisholm E, Briand LC (2008) A realistic empirical evaluation of the costs and benefits of UML in software maintenance. IEEE Trans Softw Eng 34 (3):407–432. https://doi.org/10.1109/TSE.2008.15

    Article  Google Scholar 

  • Feitelson DG (2015) Using students as experimental subjects in software engineering research - A review and discussion of the evidence, CoRR, vol. abs/1512.08409. [Online]. Available: 1512.08409

  • Fowler M (2003) UML distilled: a brief guide to the standard object modeling language, 3rd edn. Addison-Wesley Longman Publishing Co., Inc., Boston

    Google Scholar 

  • Grissom RJ, Kim JJ (2005) Effect sizes for research: A broad practical approach, 2nd edn. Lawrence Earlbaum Associates, New Jersey

    Google Scholar 

  • Gurov V, Mazin M, Narvsky A, Shalyto A (2007) Tools for support of automata-based programming. Programm Comput Softw 33:343–355

    Article  MATH  Google Scholar 

  • Hamon G (2005) A denotational semantics for stateflow. In: Proceedings of the 5th ACM International Conference on Embedded Software, ser. EMSOFT ’05. ACM, New York, pp 164–172. [Online]. Available: https://doi.org/10.1145/1086228.1086260

  • Hamon G, Rushby J (2007) An operational semantics for stateflow. Int J Softw Tools Technol Transf 9(5):447–456. [Online]. Available: https://doi.org/10.1007/s10009-007-0049-7

    Article  MATH  Google Scholar 

  • Harel D (1987) Statecharts: a visual formalism for complex systems. S Comput Programm 8(3):231–274. [Online]. Available: http://www.sciencedirect.com/science/article/pii/0167642387900359

    Article  MathSciNet  MATH  Google Scholar 

  • Heiberger R, Robbins N (2014) Design of diverging stacked bar charts for likert scales and other applications. J Stat Softw 57(5):1–32, 4. [Online]. Available: http://www.jstatsoft.org/v57/i05

    Article  Google Scholar 

  • Hutchinson J., Whittle J, Rouncefield M, Kristoffersen S (2011) Empirical assessment of MDE in industry. In: Proceedings of 33rd International Conference on Software Engineering (ICSE 2011). ACM, New York, pp 471–480

  • Hovsepyan A, Scandariato R, Van Baelen S, Joosen W, Demeyer S (2011) Preserving aspects via automation: A maintainability study. In: Proceedings of 5th International Symposium on Empirical Software Engineering and Measurement (ESEM 2011). IEEE CS, Washington, pp 315–324. [Online]. Available: https://doi.org/10.1109/ESEM.2011.40

  • Jedlitschka A, Ciolkowski M, Pfahl D (2008) Reporting experiments in software engineering. In: Guide to Advanced Empirical Software Engineering. Springer, Berlin

    Chapter  Google Scholar 

  • Kapteijns T, Jansen S, Brinkkemper S, Houet H, Barendse R (2009) A comparative case study of model driven development vs traditional development: the tortoise or the hare. In: 4th European Workshop on “From code centric to model centric software engineering: Practices, Implications and ROI” (C2M 2009). CTIT Workshop Proceedings Series, pp 22–33

  • Kitchenham B, Al-Khilidar H, Babar M, Berry M, Cox K, Keung J, Kurniawati F, Staples M, Zhang H, Zhu L (2008) Evaluating guidelines for reporting empirical software engineering studies. Empir Softw Eng 13:97–121

    Article  Google Scholar 

  • Kleppe AG, Warmer J, Bast W (2003) MDA explained: the model driven architecture: practice and promise. Addison-Wesley Longman Publishing Co. Inc, Boston

    Google Scholar 

  • Leotta M, Ricca F, Antoniol G, Garousi V, Zhi J, Ruhe G (2013) A pilot experiment to quantify the effect of documentation accuracy on maintenance tasks. In: Proceedings of 29th International Conference on Software Maintenance (ICSM 2013). IEEE, pp 428–431. https://doi.org/10.1109/ICSM.2013.64

  • Liebel G, Marko N, Tichy M, Leitner A, Hansson J (2014) Assessing the state-of-practice of model-based engineering in the embedded systems domain. In: Dingel J, Schulte W, Ramos I, Abrahão S, Insfran E (eds) Proceedings of 17th International Conference on Model Driven Engineering Languages and Systems (MODELS 2014), ser. Lecture Notes in Computer Science, vol 8767. Springer, Berlin, pp 166–182. [Online]. Available: https://doi.org/10.1007/978-3-319-11653-2_11

    Google Scholar 

  • Martínez Y, Cachero C, Matera M, Abrahao S, Luján S (2011) Impact of MDE approaches on the maintainability of web applications: an experimental evaluation. In: Proceedings of 30th International Conference on Conceptual Modeling (ER 2011). Springer-Verlag, Berlin, pp 233–246. [Online]. Available: http://dl.acm.org/citation.cfm?id=2075144.2075168

    Chapter  Google Scholar 

  • Martínez Y, Cachero C, Meliá S (2014) Empirical study on the maintainability of web applications: Model-driven engineering vs code-centric. Empir Softw Eng 19 (6):1887–1920. [Online]. Available: https://doi.org/10.1007/s10664-013-9269-5

    Article  Google Scholar 

  • Maraninchi F, Rémond Y (2001) Argos: an automaton-based synchronous language. Comput Lang 27(1–3):61–92. visual Formal Methods-VFM’99 Symposium. [Online]. Available: http://www.sciencedirect.com/science/article/pii/S0096055101000169

    Article  MATH  Google Scholar 

  • Mellegård N, Ferwerda A, Lind K, Heldal R, Chaudron MRV (2016) Impact of introducing domain-specific modelling in software maintenance: An industrial case study. IEEE Trans Softw Eng 42(3):245–260

    Article  Google Scholar 

  • Melia S, Gomez J, Perez S, Diaz O (2008) A model-driven development for gwt-based rich internet applications with ooh4ria. In: Proceedings of 8th International Conference on Web Engineering (ICWE 2008), pp 13–23

  • Motulsky H (2010) Intuitive biostatistics: a non-mathematical guide to statistical thinking. Oxford University Press, Oxford. [Online]. Available: http://books.google.it/books?id=R477U5bAZs4C

    Google Scholar 

  • Mussbacher G, Amyot D, Breu R, Bruel J-M, Cheng B, Collet P, Combemale B, France R, Heldal R, Hill J, Kienzle J, Schöttle M, Steimann F, Stikkolorum D, Whittle J (2014) The relevance of model-driven engineering thirty years from now. In: Dingel J., Schulte W., Ramos I., Abrahão S, Insfran E (eds) Proceedings of 17th International Conference on Model Driven Engineering Languages and Systems (MODELS 2014), ser. Lecture Notes in Computer Science, vol 8767. Springer, Berlin, pp 183–200. [Online]. Available: https://doi.org/10.1007/978-3-319-11653-2_12

    Google Scholar 

  • Oppenheim AN (1992) Questionnaire Design, Interviewing and Attitude Measurement. Pinter, London

    Google Scholar 

  • Papotti P, do Prado A, de Souza W, Cirilo C, Pires L (2013) A quantitative analysis of model-driven code generation through software experimentation. In: Salinesi C, Norrie M, Pastor Ó (eds) Advanced Information Systems Engineering, ser. Lecture Notes in Computer Science, vol 7908. Springer, Berlin Heidelberg, pp 321–337. [Online]. Available: https://doi.org/10.1007/978-3-642-38709-8_21

    Google Scholar 

  • Reggio G, Leotta M, Ricca F (2014) Who knows/uses what of the UML: A personal opinion survey. In: Dingel J., Schulte W, Ramos I., Abrahão S, Insfran E (eds) Proceedings of 17th International Conference on Model Driven Engineering Languages and Systems (MODELS 2014), ser. Lecture Notes in Computer Science, vol 8767. Springer, Berlin, pp 149–165. https://doi.org/10.1007/978-3-319-11653-2_10

    Google Scholar 

  • Reggio G, Leotta M, Ricca F, Clerissi D (2015) What are the used UML diagram constructs? A document and tool analysis study covering activity and use case diagrams. In: Hammoudi S, Pires LF, Filipe J, das Neves RC (eds) Model-Driven Engineering and Software Development, ser. Communications in Computer and Information Science, vol 506. Springer, Berlin, pp 66–83. [Online]. Available: https://doi.org/10.1007/978-3-319-25156-1_5

    Google Scholar 

  • Pfleeger SL, Menezes W (2000) Marketing technology to software practitioners. IEEE Softw 17(1):27–33

    Article  Google Scholar 

  • Ricca F, Di Penta M, Torchiano M, Tonella P, Ceccato M, Visaggio CA (2008) Are Fit tables really talking?: a series of experiments to understand whether fit tables are useful during evolution tasks. In: Proceedings of 30th International Conference on Software Engineering (ICSE 2008). ACM, New York, pp 361–370. [Online]. Available: https://doi.org/10.1145/1368088.1368138

  • Ricca F, Di Penta M, Torchiano M, Tonella P, Ceccato M (2010) How developers’ experience and ability influence web application comprehension tasks supported by UML stereotypes: A series of four experiments. IEEE Trans Softw Eng 36:96–118

    Article  Google Scholar 

  • Ricca F, Leotta M, Reggio G, Tiso A, Guerrini G, Torchiano M (2012) Using UniMod for maintenance tasks: An experimental assessment in the context of model driven development. In: Proceedings of 4th International Workshop on Modeling in Software Engineering (MiSE 2012). IEEE, pp 77–83. [Online]. Available: https://doi.org/10.1109/MISE.2012.6226018

  • R Core Team R (2013) A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna. ISBN 3-900051-07-0. [Online]. Available: http://www.R-project.org/

    Google Scholar 

  • Sheskin D (2007) Handbook of parametric and nonparametric statistical procedures, 4th edn. Chapman & All, UK

    MATH  Google Scholar 

  • Shalyto AA, Tukkel NI (2001) SWITCH technology: an automated approach to developing software for reactive systems. Programm Comput Softw 27:260–276

    Article  MATH  Google Scholar 

  • Shapiro SS, Wilk MB (1965) An analysis of variance test for normality (complete samples). Biometrika 52:3

    Article  MathSciNet  MATH  Google Scholar 

  • Sommerville I (2006) Software engineering. Addison-Wesley Longman Publishing Co. Inc, Boston

    MATH  Google Scholar 

  • Stahl T, Voelter M, Czarnecki K (2006) Model-driven software development: technology, engineering, management. Wiley, New Jersey

    Google Scholar 

  • Svahnberg M, Aurum A, Wohlin C (2008) Using students as subjects-an empirical evaluation. In: Proceedings of 2nd International Symposium on Empirical Software Engineering and Measurement (ESEM 2008), pp 288–290

  • Torchiano M, Tomassetti FCA, Ricca F, Tiso A, Reggio G (2013) Relevance, benefits, and problems of software modelling and model driven techniques–a survey in the Italian industry. J Syst Softw 86(8):2110–2126

    Article  Google Scholar 

  • Torchiano M (2015) effsize: Efficient effect size computation. r package version 0.5.5. [Online]. Available: http://CRAN.R-project.org/package=effsize

  • Torchiano M, Scanniello G, Ricca F, Reggio G, Leotta M (2017) Do UML object diagrams affect design comprehensibility? Results from a family of four controlled experiments. J Vis Lang Comput 41:10–21. [Online]. Available: https://doi.org/10.1016/j.jvlc.2017.06.002

    Article  Google Scholar 

  • Wheeler B (2016) lmPerm: Permutation tests for linear models. R package version 2.0. [Online]. Available: http://CRAN.R-project.org/package=lmPerm

  • Whittle J, Hutchinson J, Rouncefield M (2014) The state of practice in model-driven engineering. IEEE Softw 31(3):79–85

    Article  Google Scholar 

  • Wohlin C, Runeson P, Höst M, Ohlsson M, Regnell B, Wesslén A (2012) Experimentation in software engineering. Springer-Verlag, Berlin Heidelberg

    Book  MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. DIBRIS, Università di Genova, Genova, Italy

    Filippo Ricca, Maurizio Leotta, Alessandro Tiso, Giovanna Guerrini & Gianna Reggio

  2. DAUIN, Politecnico di Torino, Torino, Italy

    Marco Torchiano

Authors
  1. Filippo Ricca
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Marco Torchiano
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Maurizio Leotta
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Alessandro Tiso
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Giovanna Guerrini
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Gianna Reggio
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Maurizio Leotta.

Additional information

Communicated by: Michel R. V. Chaudron

Appendix

Appendix

This Appendix reports detailed analysis results. They are not essential to understand the paper, though they provide additional information and complement the results provided in the main body of the papers.

1.1 A.1 Detailed Hypotheses Testing

We report here the Heatmap graph concerning the hypotheses tested at the individual maintenance task (MT) level. In particular, Fig. 17 reports the p-values of the Mann-Whitney tests; Fig. 18 shows the Cliff’s delta values.

Fig. 17
figure 17

Heat map of MW p-values

Full size image
Fig. 18
figure 18

Heat map of Cliff d

Full size image

1.2 A.2 Influence of Lab Order

As mentioned in Section 5.3, we performed a set of two-way permutation tests to check the effect of Lab order on the dependent variables. The results of the tests are presented in Table 19 for TotalEfficiency, Table 20 for TotalTime, and Table 21 for TotalCorrectness.

Table 19 Permutation test of TotalEfficiency vs. Experiment and Lab order
Full size table
Table 20 Permutation test of TotalTime vs. Experiment and Lab order
Full size table
Table 21 Permutation test of TotalCorrectness vs. Experiment and Lab order
Full size table

The tests consistently show neither a main effect from the Lab order nor any interaction with the experiment.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ricca, F., Torchiano, M., Leotta, M. et al. On the impact of state-based model-driven development on maintainability: a family of experiments using UniMod. Empir Software Eng 23, 1743–1790 (2018). https://doi.org/10.1007/s10664-017-9563-8

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10664-017-9563-8

Keywords

Search

Navigation